Kategorie: Kubernetes

Docker Donates the cnab-to-oci Library to cnab.io

Docker is proud and happy to announce the donation of our cnab-to-oci library to the CNAB project . This project was created last year after Microsoft and Docker moved the CNAB specification to the Linux Foundation’s Joint Development Foundation. At that time, the CNAB specification repository was moved from the deislab GitHub organization to the new cnabio organization. The reference implementations – cnab-go which is the Golang library implementation of the specification and duffle which is the CLI reference implementation – were also moved.

What is cnab-to-oci for?

Docker helped with the development of the CNAB specification and its reference implementations, and led the work on the cnab-to-oci library for sharing a CNAB bundle using an existing container registry. This library is now used by 3 CNAB tools, Docker App, Porter and duffle, as well as Docker Hub. It successfully demonstrated how to push, pull and share a CNAB bundle using a registry. This work will be used as a foundation for the future CNAB Registries specification.

The transfer is already in effect, so starting now please refer to github.com/cnabio/cnab-to-oci in your Golang imports.

How does cnab-to-oci store a CNAB bundle into a registry?

As you may know, the OCI image specification introduces two main objects: the OCI Manifest and the OCI Image Index. The first one is well known and represents the classic Docker image. The other one was, at first, used to store multi-architecture images (see nginx as an example).

But what you may not know is that the specification doesn’t restrict the use of OCI Indexes to multi-arch images. You can store almost anything you want, as long as you meet the specification, and it is quite open.

cnab-to-oci uses this openness to push the bundle.json, but also the invocation image and the component images (or service images for a Docker App). It pushes everything in the same repository, so one has the guarantee that when someone pulls her/his bundle, all the components can be pulled as well.

Demo Time

While cnab-to-oci is implemented as a library that can be used by other tools, the repository contains a handy CLI tool that can perform push and pull of any CNAB bundle.json.

With the following command we push a bundle example to the Docker Hub repository. It pushes all the manifests found in the bundle, then creates an OCI Index and pushes it at the end. The digest we get as a result is pointing to the OCI Index of the bundle.

$ make bin/cnab-to-oci…$ ./bin/cnab-to-oci push examples/helloworld-cnab/bundle.json -t hubusername/repo:demo –log-level=debug –auto-update-bundleDEBU[0000] Fixing up bundle docker.io/hubusername/repo:demoDEBU[0000] Updating entry in relocation map for “cnab/helloworld:0.1.1”Starting to copy image cnab/helloworld:0.1.1…Completed image cnab/helloworld:0.1.1 copyDEBU[0004] Bundle fixedDEBU[0004] Pushing CNAB Bundle docker.io/hubusername/repo:demoDEBU[0004] Pushing CNAB Bundle ConfigDEBU[0004] Trying to push CNAB Bundle ConfigDEBU[0004] CNAB Bundle Config DescriptorDEBU[0004] {  “mediaType”: “application/vnd.cnab.config.v1+json”,  “digest”: “sha256:e91b9dfcbbb3b88bac94726f276b89de46e4460b55f6e6d6f876e666b150ec5b”,  “size”: 498}DEBU[0005] Trying to push CNAB Bundle Config ManifestDEBU[0005] CNAB Bundle Config Manifest DescriptorDEBU[0005] {  “mediaType”: “application/vnd.oci.image.manifest.v1+json”,  “digest”: “sha256:6ec4fd695cace0e3d4305838fdf9fcd646798d3fea42b3abb28c117f903a6a5f”,  “size”: 188}DEBU[0006] Failed to push CNAB Bundle Config Manifest, trying with a fallback methodDEBU[0006] Trying to push CNAB Bundle ConfigDEBU[0006] CNAB Bundle Config DescriptorDEBU[0006] {  “mediaType”: “application/vnd.oci.image.config.v1+json”,  “digest”: “sha256:e91b9dfcbbb3b88bac94726f276b89de46e4460b55f6e6d6f876e666b150ec5b”,  “size”: 498}DEBU[0006] Trying to push CNAB Bundle Config ManifestDEBU[0006] CNAB Bundle Config Manifest DescriptorDEBU[0006] {  “mediaType”: “application/vnd.oci.image.manifest.v1+json”,  “digest”: “sha256:b9616da7500f8c7c9a5e8d915714cd02d11bcc71ff5b4fd190bb77b1355c8549”,  “size”: 193}DEBU[0006] CNAB Bundle Config pushedDEBU[0006] Pushing CNAB IndexDEBU[0006] Trying to push OCI IndexDEBU[0006] {“schemaVersion”:2,”manifests”:[{“mediaType”:”application/vnd.oci.image.manifest.v1+json”,”digest”:”sha256:b9616da7500f8c7c9a5e8d915714cd02d11bcc71ff5b4fd190bb77b1355c8549″,”size”:193,”annotations”:{“io.cnab.manifest.type”:”config”}},{“mediaType”:”application/vnd.docker.distribution.manifest.v2+json”,”digest”:”sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6″,”size”:942,”annotations”:{“io.cnab.manifest.type”:”invocation”}}],”annotations”:{“io.cnab.keywords”:”[”helloworld”,”cnab”,”tutorial”]”,”io.cnab.runtime_version”:”v1.0.0″,”org.opencontainers.artifactType”:”application/vnd.cnab.manifest.v1″,”org.opencontainers.image.authors”:”[{”name”:”Jane Doe”,”email”:”jane.doe@example.com”,”url”:”https://example.com”}]”,”org.opencontainers.image.description”:”A short description of your bundle”,”org.opencontainers.image.title”:”helloworld”,”org.opencontainers.image.version”:”0.1.1″}}DEBU[0006] OCI Index DescriptorDEBU[0006] {  “mediaType”: “application/vnd.oci.image.index.v1+json”,  “digest”: “sha256:fcee8577f3acc8ddc6e0280e6d1eb15be70bdff460fe7353abf917a872487af2”,  “size”: 926}DEBU[0007] CNAB Index pushedDEBU[0007] CNAB Bundle pushedPushed successfully, with digest “sha256:fcee8577f3acc8ddc6e0280e6d1eb15be70bdff460fe7353abf917a872487af2”

Let’s check that our bundle has been pushed on Docker Hub:

We can now pull our bundle back from the registry. It will only fetch the bundle.json file, but as you may notice this now has a digested reference for the image manifest of every component, inside the same registry repository. The Docker Engine will pull any images required by the bundle at runtime. So pulling a bundle is a lightweight process.

$ ./bin/cnab-to-oci pull hubusername/repo:demo –log-level=debugDEBU[0000] Pulling CNAB Bundle docker.io/hubusername/repo:demoDEBU[0000] Getting OCI Index DescriptorDEBU[0001] {  “mediaType”: “application/vnd.oci.image.index.v1+json”,  “digest”: “sha256:fcee8577f3acc8ddc6e0280e6d1eb15be70bdff460fe7353abf917a872487af2”,  “size”: 926}DEBU[0001] Fetching OCI Index sha256:fcee8577f3acc8ddc6e0280e6d1eb15be70bdff460fe7353abf917a872487af2DEBU[0001] {  “schemaVersion”: 2,  “manifests”: [    {      “mediaType”: “application/vnd.oci.image.manifest.v1+json”,      “digest”: “sha256:b9616da7500f8c7c9a5e8d915714cd02d11bcc71ff5b4fd190bb77b1355c8549”,      “size”: 193,      “annotations”: {        “io.cnab.manifest.type”: “config”      }    },    {      “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,      “digest”: “sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6”,      “size”: 942,      “annotations”: {        “io.cnab.manifest.type”: “invocation”      }    }  ],  “annotations”: {    “io.cnab.keywords”: “[”helloworld”,”cnab”,”tutorial”]”,    “io.cnab.runtime_version”: “v1.0.0”,    “org.opencontainers.artifactType”: “application/vnd.cnab.manifest.v1”,    “org.opencontainers.image.authors”: “[{”name”:”Jane Doe”,”email”:”jane.doe@example.com”,”url”:”https://example.com”}]”,    “org.opencontainers.image.description”: “A short description of your bundle”,    “org.opencontainers.image.title”: “helloworld”,    “org.opencontainers.image.version”: “0.1.1”  }}DEBU[0001] Getting Bundle Config Manifest DescriptorDEBU[0001] {  “mediaType”: “application/vnd.oci.image.manifest.v1+json”,  “digest”: “sha256:b9616da7500f8c7c9a5e8d915714cd02d11bcc71ff5b4fd190bb77b1355c8549”,  “size”: 193,  “annotations”: {    “io.cnab.manifest.type”: “config”  }}DEBU[0001] Getting Bundle Config Manifest sha256:b9616da7500f8c7c9a5e8d915714cd02d11bcc71ff5b4fd190bb77b1355c8549DEBU[0001] {  “schemaVersion”: 2,  “config”: {    “mediaType”: “application/vnd.oci.image.config.v1+json”,    “digest”: “sha256:e91b9dfcbbb3b88bac94726f276b89de46e4460b55f6e6d6f876e666b150ec5b”,    “size”: 498  },  “layers”: null}DEBU[0001] Fetching Bundle sha256:e91b9dfcbbb3b88bac94726f276b89de46e4460b55f6e6d6f876e666b150ec5bDEBU[0002] {  “schemaVersion”: “v1.0.0”,  “name”: “helloworld”,  “version”: “0.1.1”,  “description”: “A short description of your bundle”,  “keywords”: [    “helloworld”,    “cnab”,    “tutorial”  ],  “maintainers”: [    {      “name”: “Jane Doe”,      “email”: “jane.doe@example.com”,      “url”: “https://example.com”    }  ],  “invocationImages”: [    {      “imageType”: “docker”,      “image”: “cnab/helloworld:0.1.1”,      “contentDigest”: “sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6”,      “size”: 942,      “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”    }  ]}

cnab-to-oci has been integrated with Docker App in the last beta release v0.9.0-beta1, to let you push and pull your entire application with the same UX as pushing a regular Docker container image. As Docker App is a standard CNAB runtime, it can also run this generic CNAB example:

$ docker app pull hubusername/repo:demoSuccessfully pulled “helloworld” (0.1.1) from docker.io/hubusername/repo:demo$ docker app run hubusername/repo:demoPort parameter was set to Install actionAction install complete for upbeat_nobelApp “upbeat_nobel” running on context “default”

Want to Know More?

If you’re interested in getting more details about CNAB, a few blog posts are available:

Multi-arch All The ThingsBuilding Multi-Arch Images for Arm and x86 with Docker DesktopAnnouncing CNABDocker App and CNABNext Steps for Cloud Native Application Bundles

Please note that we will give a talk about this topic at KubeCon Europe 2020: “Sharing is Caring! Push your Cloud Application to an OCI Registry – Silvin Lubecki & Djordje Lukic”

And of course, you can also find more information directly on the cnab-to-oci GitHub repository.

Contributions are welcome!!!
The post Docker Donates the cnab-to-oci Library to cnab.io appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Hack Week: How Docker Drives Innovation from the Inside

Since its founding, Docker’s mission has been to help developers bring their ideas to life by conquering the complexity of app development. With millions of Docker developers worldwide, Docker is the de facto standard for building and sharing containerized apps. 

So what is one source of ideas we use to simplify the lives of developers? It starts with being a company of software developers who builds products for software developers. One of the more creative ways Docker has been driving innovation internally is through hackathons. These hackathons have proven to be a great platform for Docker employees to showcase their talent and provide unique opportunities for teams across Docker’s business functions to come together. Our employees get to have fun while creating solutions to problems that simplify the lives of Docker developers.

At Docker, our engineers are always looking for ways to improve their own workflows so as to ship quality code faster. Hack Week gives us a chance to explore the boundaries of what’s possible, and the winning ‘hacks’ make their way into our products to benefit our global developer community.

-Scott Johnston, Docker CEO

With that context, let’s break down how Docker runs employee hackathons. Docker is an open source company, and in the spirit of openness, I am sharing all the gory details here of our hackathon. 

First of all, our hackathon is known as “Hack Week.” We conduct hackathons twice a year. Docker uses Slack channels to manage employee communications, Confluence for team workspaces and Zoom for video conferencing and recording of demos. For example, we have a Confluence Hack Week site with all the info an employee needs to participate: hackathon rules, team sign-ups, calendar and schedule, demo recordings and results.

Because we still need to perform our day jobs, we run Hack Week for a full work week where employees can manage their time but are granted 20% of that time to work on their hackathon project during work hours. Below is a screenshot of Docker’s internal site for Hack Week that provides simple guidance and voting criteria – every employee gets a vote!

Docker Hackathon Home Page

What makes this fun at Docker is the fact that we have employees participating from Paris, Cambridge (UK) and San Francisco. There are no constraints on how teams form. You can have members from all three locations form as one team. Signing up is simple – all we require is a team name, your team members, your region and a 1-3 sentence description of your “hack.” Below is the calendar from Docker’s last Hack Week which we ran back in December 2019. This should give you a good overview of how we execute Hack Week. This actually runs quite smoothly for Docker despite the 8-9 hour time difference between our teams in San Francisco and the teams in the UK and France. 

The winning team for December’s Hack Week was Team M&Ms (s/o to Mathieu Champion in Paris and Michael Parker in Cambridge) after garnering the most employee votes. The description of their hack was “run everything from Docker Desktop.” The hack enables auto-generation of dockerfiles from Docker Desktop. (A dockerfile is a text document that contains all the commands a user could call on the command line to assemble a container image). 

I spoke with Michael Parker regarding his motivations for participation in Hack Week. “Hack Week is a great innovation platform – it lets employees show what can be easily implemented with our current systems and dream a bit bigger about what might be possible rather than focusing on the incremental feature tweaks and bug fixes.” 

Finally, I have shared the recorded video below from our Hack Week winning team. This will give you a good idea as to how we present, collaborate and vote in a virtual work environment with teams spread across two continents and an island. It’s a 6-minute video and will give you a great view of how passionate we are about making the lives of developers that much better by making their jobs that much easier and productive.

Feel free to let any of this content we have shared inspire your organization’s employees to plan and conduct your own hackathons. I remember back in 2012 when I was participating in a public hackathon at Austin’s SXSW Interactive Conference seeing none other than Deepak Chopra kicking off the event and inspiring developers. He talked about hackathons as a form of “creative chaos” and how conflict and destruction of established patterns can often lead to creativity and innovation. I think this is a great description of a hackathon. Are you ready for some creative chaos inside your own organization?

The post Hack Week: How Docker Drives Innovation from the Inside appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Our Favourite Picks from the KubeCon Europe 2020 Schedule

Last Wednesday, the CNCF released the KubeCon Europe 2020 schedule. There are so many talks at KubeCon it can be daunting even to decide what to go to see! Here are some talks by the team at Docker, and some others we think will be particularly interesting. Looking forward to seeing you in Amsterdam!

Simplify Your Cloud-Native Application Packaging and Deployments – Chris Crone

Chris is an engineer in our Paris office and is also co-executive director of the CNAB project. CNAB (Cloud Native Application Bundle) is a specification for bundling up cloud-native applications, which can consist of multiple containers, into a single object that can be pushed to a registry. Open source projects using CNAB, like Docker App or Porter allow you to package apps that would normally require multiple tools like Terraform, Helm, and shell to deploy, into a single tooling agnostic packaging format. These packages can then be shared using existing container registries and used with other CNAB compliant tools. This can really simplify cloud-native development.

Sharing is Caring! Push your Cloud Application to an OCI Registry – Silvin Lubecki & Djordje Lukic

Did you know that you can store anything into a container registry? Did you ever wonder what black magic is behind multi-architecture images? The OCI Image specification is a standard purposely generic enough to enable use cases other than “just” container images.

This talk will give an overview of how images in registries work, and how you can push CNAB applications and other custom resources into a registry. It will also cover our battle scars with the different interpretations of the OCI spec by the mainstream registries. 

How to Work in Cloud Native Security: Demystifying the Security Role – Justin Cormack, Docker

Working in security can be intimidating and the shortage of people in the space makes hiring difficult. But especially in cloud-native environments, security is something everyone must own. If you’ve ever asked yourself, “what does it take to work in security in a cloud-native environment? How can you move into security from a dev or an ops position? Where should you start and what should you learn about?” then this talk is for you. I decided to submit this talk as my journey into working in security was fairly accidental, and I realised that this is true for many people. I meet a lot of people interested in getting into security, through the CNCF SIG Security and elsewhere, and hope I can give help and encouragement.

More interesting talks

I wrote about the work the community is doing in the CNCF on Notary v2 last week. If you found this interesting and want to learn more, we have an introductory session, with me and Omar Paul from Amazon, which will give a beginner’s view and a working session for in-depth work with Steve Lasker from Microsoft and me.

If you want even more on container signing, Justin Cappos and Lukas Puehringer from New York University have a session on securing container delivery with TUF and another on supply chain security with in-toto.

The containerd community continues to grow and innovate. Phil Estes from IBM and Derek McGowan from Docker are covering the Introduction to containerd, while Akihiro Suda and Wei Fu are doing the containerd deep dive. Also on the containerd theme, the great teachers Bret Fisher and Jerome Petazzoni are giving a tutorial: Kubernetes Runtimes: Translating your Docker skills to containerd.

Dominique Top and Ivan Pedrazas run the London Docker meetup and are both lovely people who have built up a great community. Learn from them with 5 Things you Could do to Improve your Local Community.

Lastly, my friend Lee Calcotte always gives great talks, and this one about how to understand the details of traffic control appeals to my geek side: Discreetly Studying the Effects of Individual Traffic Control Functions.
The post Our Favourite Picks from the KubeCon Europe 2020 Schedule appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Introducing the Docker Index: Insight from the World’s Most Popular Container Registry

8 billion pulls! Yes, that’s billion with a B! This number represents a little known level of activity and innovation happening across the community and ecosystem, all in just one average month. How do we know? From the number of pulls and most popular images to top architectures, data from Docker Hub and Docker Desktop provide a window into application development trends in the age of containers. 

Today, we are sharing these findings in something we call the Docker Index – a look at developers’ preferences and trends, as told by using anonymized data from five million Docker Hub and two million Docker Desktop users, as well as countless other developers engaging with content on Hub. 

At Docker, we’re always looking for ways to make life easier for developers. Understanding the what, why and how behind these projects is imperative. As these trends evolve, we will continue to share updates on the findings.

Whether containers will become mainstream is no longer a topic of debate. As the Docker Index data suggests, containers have become a mainstay to how modern, distributed apps are built and shared so they can run anywhere. 

Usage is showing no signs of slowing down. Docker Desktop and Docker Hub are reaching an increasing number of developers and users are engaging with content from Hub at higher rates. Content from community developers and open source projects continues to make Hub a central and valuable source for developers looking to build containerized applications. 

Collaboration is key when building apps so that developers aren’t starting from scratch. Containers have helped to make building blocks the new norm. With container images readily accessible and shareable, everyone can be more productive. 

Modern apps also give rise to increasingly diverse development environments, drawing more attention to the importance of choice. The ability to select your preferred framework, operating system and architecture go a long way in creating a more productive experience for modern app development.

The ecosystem and community are shaping the future of software development and containers are at the heart of this transformation. The level of activity and collaboration is hitting a new gear and with it, continued advancements in how developers build and share apps. We look forward to sharing updates on the Docker Index data over the course of this year. 

To get started with Docker, download Docker Desktop and take a tutorial here https://www.docker.com/get-started.
The post Introducing the Docker Index: Insight from the World’s Most Popular Container Registry appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

How We Solved a Report on docker-compose Performance on macOS Catalina

Photo by Caspar Camille Rubin on Unsplash

As a Docker Compose maintainer, my daily duty is to check for newly reported issues and try to help users through misunderstanding and possible underlying bugs. Sometimes issues are very well documented, sometimes they are nothing much but some “please help” message. And sometimes they look really weird and can result in funny investigations. Here is the story of how we solved one such report…

A one-line bug report

An issue was reported as “docker-compose super slow on macOS Catalina” – no version, no details. How should I prioritize this? I don’t even know if the reporter is using the latest version of the tool – the opened issue doesn’t follow the bug reporting template. This is just a one-liner. But for some reason, I decided to take a look at it anyway and diagnose the issue.

Without any obvious explanation for super-slowness, I decided to take a risk and upgrade my own MacBook to OSX Catalina. I was able to reproduce significant slow down in docker-compose execution, waiting seconds for the very first line printed on the console even to display usage on invalid command.

Investigating the issue

In the meantime, some users reported getting correct performance when installing docker-compose as a plain python software, not with the packaged executable. The docker-compose executable is packaged using PyInstaller, which embeds a Python runtime and libraries with application code in a single executable file. As a result, one gets a distributable binary that can be created for Windows, Linux and OSX.  I wrote a minimalist “hello world” python application and was able to reproduce the same weird behaviour once packaged the same way docker-compose is, i.e. a few second startup delay.

Here comes the funny part. I’m a remote worker on the Docker team, and I sometimes have trouble with my Internet connection. It happened this exact day, as my network router had to reboot. And during the reboot sequence, docker-compose performance suddenly became quite good … but eventually, the initial execution delay came back. How do you explain such a thing?

So I installed Charles proxy to analyze network traffic, and discovered a request sent to api.apple-cloudkit.com each and everytime docker-compose was run. Apple Cloudkit is Apple cloud storage SDK, and there’s no obvious relation between docker-compose and this service.

As the Docker Desktop team was investigating Catalina support during this period, I heard about the notarization constraints introduced by the Apple OS upgrade. I decided to reconfigure my system with system integrity check disabled (you have to run ‘csrutil disable’ from recovery console on boot). Here again, docker-compose suddenly went reasonably fast.

Looking into PyInstaller implementation details, when executed docker-compose binary extracts itself into a temporary folder, then executes the embedded Python runtime to run the packaged application. This bootstrap sequence takes a blink of an eye on a recent computer with tmp folder mapped to memory, but on my Catalina-upgraded MacBook it took up to 10 seconds – until I disabled integrity check.

Confirming the hypothesis

My assumption was: OSX Catalina reinforced security constraints do apply to the python runtime as it gets extracted, as a security scan, and the system does send a scan report to apple over its own cloud storage service. I can’t remember having approved sending such data to Apple, but I admit I didn’t carefully read the upgrade guide and service agreement before I hit the “upgrade to Catalina” button. As a fresh new Python runtime is extracted for temporary execution, this takes place each and every time we run a docker-compose command: new system scan, new report sent to apple – not even as a background task. 

To confirm this hypothesis, I built a custom flavour of docker-compose using an alternate PyInstaller configuration, so it doesn’t create a single binary, but a folder with runtime and libraries. The first execution of this custom docker-compose packaging took 10 seconds again (initial scan by the system), but subsequent commands were as efficient as expected.

The resolution

A few weeks later, a release candidate build was included in the Docker Desktop Edge channel to confirm that Catalina users get good performance using this alternate packaging, while not introducing unexpected bugs. Docker-compose 1.25.1 was released one month later with the bug fix confirmed. Starting with this release, docker-compose is available both as single binary packaging and as a tar.gz for OSX Catalina.

The post How We Solved a Report on docker-compose Performance on macOS Catalina appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

January Virtual Meetup Recap: Improve Image Builds Using the Features in BuildKit

This is a guest post by Docker Captain Nicholas Dille, a blogger, speaker and author with 15 years of experience in virtualization and automation. He works as a DevOps Engineer at Haufe Group, a digital media company located in Freiburg, Germany. He is also a Microsoft Most Valuable Professional.

In this virtual meetup, I share how to improve image builds using the features in BuildKit. BuildKit is an alternative builder with great features like caching, concurrency and the ability to separate your image build into multiple stages – which is useful for separating the build environment from the runtime environment. 

The default builder in Docker is the legacy builder. This is recommended for use when you need support for Windows. However, in nearly every other case, using BuildKit is recommended because of the fast build time, ability to use custom BuildKit front-ends, building stages in parallel and other features.

Catch the full replay below and view the slides to learn:

Build cache in BuildKit – instead of relying on a locally present image, buildkit will pull the appropriate layers of the previous image from a registry.How BuildKit helps prevent disclosure of credentials by allowing files to be mounted into the build process. They are kept in memory and are not written to the image layers.How BuildKit supports access to remote systems through SSH by mounting the SSH agent socket into the build without adding the SSH private key to the image.How to use the CLI plugin buildx to cross-build images for different platforms.How using the new “docker context,” the CLI is able to manage connection to multiple instances of the Docker engine. Note that it supported SSH remoting to Docker Engine.And finally, a tip that extends beyond image builds: When troubleshooting a running container, a debugging container can be started sharing the network and PID namespace. This allows debugging without changing the misbehaving container.

I also covered a few tools that I use in my workflow, namely:

goss, which allows images to be tested to match a configuration expressed in YAML. It comes with a nice wrapper called `dgoss` to use it with Docker easily. And it even provides a health endpoint to integrate into your imagetrivy, an OS tool from AquaSecurity that scans images for known vulnerabilities in the OS as well as well-known package managers.

And finally, answered some of your questions:

Why not use BuildKit by default? 

If your workflow involves building images often, then we recommend that you do set BuildKit as the default builder. Here is how to enable BuildKit by default in the docker daemon config. 

Does docker-compose work with BuildKit? 

Support for BuildKit was added in docker-compose 1.25.0 which can be enabled by setting DOCKER_BUILDKIT=1 and COMPOSE_DOCKER_CLI_BUILD=1.

What are the benefits of using BuildKit? 

In addition to the features presented, BuildKit also improves build performance in many cases.

When would I use BuildKit Secrets? (A special thank you to Captain Brandon Mitchell for answering this question)

BuildKit secrets are a good way to use a secret at build time, without saving the secret in the image. Think of it as pulling a private git repo without saving your ssh key to the image. For runtime, it’s often different compose files to support compose vs swarm mode, each mounting the secret a different way, i.e. a volume vs. swarm secret.

How do I enable BuildKit for Jenkins Docker build plugin? 

The only reference to BuildKit I was able to find refers to adding support in the Docker Pipeline plugin.

Does BuildKit share the build cache with the legacy builder? 

No, the caches are separate.

What are your thoughts on having the testing step as a stage in a multi-stage build? 

The test step can be a separate stage in the build. If the test step requires a special tool to be installed, it can be a second final stage. If your multi-stage build increases in complexity, take a look at CI/CD tools.

How does pulling the previous image save time over just doing the build? The download can be significantly faster than redoing all the work.

Is the created image still “identical” or is there any real difference in the final image artifact? 

The legacy builder, as well as BuildKit, produces identical (or rather equivalent) images.

Will Docker inspect show that the image was built using BuildKit? 

No.

Do you know any combination of debugging with docker images/containers (I use the following technologies: python and Django and Pycharm)?

No. Anyone have any advice here? 

Is Docker BuildKit supported with maven Dockerfile plugin? 

If the question is referring to Spotify’s Dockerfile Maven plugin (which is unmaintained), the answer is no. Other plugins may be able to use BuildKit when providing the environment variable DOCKER_BUILDKIT=1. Instead of changing the way the client works, you could configure the daemon to use BuildKit by default (see first question above).

What do you think about CRI-O? 

I think that containerd has gained more visibility and has been adopted by many cloud providers as the runtime in Kubernetes offerings. But I have no experience myself with CRI-O.

To be notified of upcoming meetups, join the Docker Virtual Meetup Group using your Docker ID or on Meetup.com.
The post January Virtual Meetup Recap: Improve Image Builds Using the Features in BuildKit appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Community Collaboration on Notary v2

One of the most productive meetings I had KubeCon in San Diego last November was a meeting with Docker, Amazon and Microsoft to plan a collaboration around a new version of the CNCF project Notary. We held the Notary v2 kickoff meeting a few weeks later in Seattle in the Amazon offices.

Emphasising that this is a cross-industry collaboration, we had eighteen people in the room (with more dialed in) from Amazon, Microsoft, Docker, IBM, Google, Red Hat, Sylabs and JFrog. This represented all the container registry providers and developers, other than the VMware Harbor developers who could unfortunately not make it in person. Unfortunately, we forgot to take a picture of everyone!

@awscloud, @GCPcloud, @Azure, @Docker, @RedHat, @jfrog collaborating on @CloudNativeFdn Notary v2 – touring the amazon spheres. Who would have thought… https://t.co/6VL3OucX0c pic.twitter.com/rNglQIO5ZM— Steve Lasker (@SteveLasker) December 15, 2019

The consensus and community are important because of the aims of Notary v2. But let’s go back a bit as some of you may not know what Notary is and what it is for.

The Notary project was originally started at Docker back in 2015 to provide a general signing infrastructure for containers based on The Update Framework (TUF), a model for package management security developed by Justin Cappos and his team at New York University. This is what supports the “docker trust” set of commands that allow signing containers, and the DOCKER_CONTENT_TRUST settings for validating signatures.

In 2017, Notary was donated to the CNCF, along with the TUF specification, to make it a cross-industry standard. It began to be shipped in other places as well as Docker Hub, including the Docker Trusted Registry (now a Mirantis product), IBM’s container registry, the Azure Container Registry, and with the Harbor project, another CNCF project. TUF also expanded its use cases, in the package management community, and in projects such as Uptane, a framework for updating firmware on automobiles.

So why a version 2 now? Part of the answer is that we learnt a lot of things about the usage of containers since 2015. Are container years like dog years? I am not sure, but a lot has happened since then, and the usage of containers has expanded enormously. I covered a lot of the reasons in-depth in my KubeCon talk:

Supply chain security – making sure that you ship what you intended to ship into production – has become increasingly important, as attacks on software supply chains have increased in recent years. Signatures are an important part of the validation needed in container supply chains.

Integrating Signatures in the Registry

The first big change that we want to make is because at present not every registry supports Notary. This means that if you use a mixture of registries, some may support signatures while others do. In addition, you cannot move signatures between registries. Both of these are related to the design of Notary as in effect a registry sidecar. While Notary shares the same authentication as a registry, it is built as a separate service, with its own database and API. 

Back when Notary was designed this did not seem so important. But now many people use, or want to use, complex registry configurations with local registries close to a production cluster, or at the cloud provider code is running on, or in an edge location which may be disconnected. The solution that we are working on is that rather than being a standalone sidecar service, signatures will be integrated into the OCI image specification and supported by all registries. The details of this are still being worked out, but this will make portability much easier, as signatures will be able to be pushed and pulled with images.

Improving Usability

The second big set of changes is around usability. The current way of signing containers and checking signatures is complex, as is the key management. One of the aims of Notary v2 is to have signatures and checking on by default where possible. There have been many issues stopping this with the current Notary, many of which are detailed in the KubeCon talk, including the large number of keys involved due to lack of hierarchy and delegation, and lack of standard interfaces for signature checking on different platforms such as Kubernetes.

If you want to learn more, there are weekly meetings on Mondays at 10 a.m. Pacific Time – see the CNCF community calendar for details. The Slack channel is #notary-v2 in the CNCF Slack. There will be two sessions at KubeCon Amsterdam, one introductory overview and state of where we are, and another deep dive working session on current issues. Hope to see you there!
The post Community Collaboration on Notary v2 appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Faster builds in Docker Compose 1.25.1 thanks to BuildKit Support

One of the most requested features for the docker-compose tool is definitely support for building using Buildkit which is an alternative builder with great capabilities, like caching, concurrency and ability to use custom BuildKit front-ends just to mention a few… Ahhh with a nice blue output! And the good news is that Docker Compose 1.25.1 – that was just released early January – includes BuildKit support!

BuildKit support for Docker Compose is actually achieved by redirecting the docker-compose build to the Docker CLI with a limited feature set.

Enabling Buildkit build

To enable this, we have to align some stars.

First, it requires that the Docker CLI binary present in your PATH:

$ whichdocker/usr/local/bin/docker

Second, docker-compose has to be run with the environment variable COMPOSE_DOCKER_CLI_BUILD set to 1 like in:

$ COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build

This instruction tells docker-compose to use the Docker CLI when executing a build. You should see the same build output, but starting with the experimental warning.

As docker-compose passes its environment variables to the Docker CLI, we can also tell the CLI to use BuildKit instead of the default builder. To accomplish that, we can execute this:

$ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build

A short video is worth a thousand words:

Please note that BuildKit support in docker-compose was initially released with Docker Compose 1.25.0. This feature is marked as experimental for now.

Want to know more?

Discover more options using docker-compose build –helpLearn more about Buildkit: docs.docker.com/develop/develop-images/build_enhancements

Share your feedback

Have nice and fast builds with docker-compose and please share your feedback with us!
The post Faster builds in Docker Compose 1.25.1 thanks to BuildKit Support appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Docker Desktop release 2.2 is here!

We are excited to announce that we released a new Docker Desktop version today! Thanks to the user feedback on the new features initially released in the Edge channel, we are now ready to publish them into Stable. 

Before getting to each feature into detail, let’s see what’s new in Docker Desktop 2.2:

WSL 2 as a technical preview, allowing access to the full system resources, improved boot time, access to Linux workspaces and improved file system performanceA new file sharing implementation for Windows, improving the developer inner loop user experienceA New Integrated Desktop Dashboard, to see at once glance your local running containers and Compose applications, and easily manage them.

WSL 2 – New architecture 

Back in July we released on Edge the technical preview of Docker Desktop for WSL 2, where we included an experimental integration of Docker running on an existing user Linux distribution. We learnt from our experience and re-architected our solution (covered in Simon’s blog) . 

This new architecture for WSL 2 allows users to: 

Use Kubernetes on the WSL 2 backendWork with just WSL 2/turn off the traditional HyperV VM while working with WSL 2Continue to work as they did in the traditional Docker Desktop with a friendly networking stack, support for http proxy settings, and trusted CA synchronization Start Docker Desktop in <5 secondsUse Linux Workspaces 

To make use of the WSL 2 features you will need to be on a Windows preview version that supports WSL 2.

Read More on WSL 2

File system improvements on Windows 

For existing Windows users not on Windows Insider builds we have been working on improving the user experience we have today for the inner loop. Traditionally with Docker Desktop on Windows, we have relied upon the Samba protocol to manage the interaction between the Linux file system working with Docker and the Windows file system. We have now replaced this with gRPC FUSE, which:

uses caching to (for example) reduce page load time in Symfony by up to 60%;supports Linux inotify events, triggering automatic recompilation / reload when the source code is changed;is independent of how you authenticate to Windows: smartcard, Azure AD are all fine;always works irrespective of whether your VPN is connected or disconnected;reduces the amount of code running as Administrator.

Read More on new File Sharing

New Integrated Desktop Dashboard

Last but not least, Docker Desktop now includes an interactive Dashboard UI for managing your local running containers and Compose applications. We have been listening to developers and working hard to incorporate a single user interface across Mac and Windows so that we could look at how we can make it easier to work with Docker locally. Historically Docker offered similar capability with Kitematic, which we plan to archive in 2020 and replace with the new Desktop Dashboard.

Read More on Desktop Dashboard

Get started today

You can try all of the new features now by getting Docker Desktop 2.2!

Download Docker Desktop 2.2 for Windows

Download Docker Desktop 2.2 for macOS

The post Docker Desktop release 2.2 is here! appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/