Kategorie: Kubernetes

How Docker Desktop Networking Works Under the Hood

Modern applications make extensive use of networks. At build time it’s common to apt-get/dnf/yum/apk install a package from a Linux distribution’s package repository. At runtime an application may wish to connect() to an internal postgres or mysql database to persist some state, while also calling listen() and accept() to expose APIs and UIs over TCP and UDP ports. Meanwhile developers need to be able to work from anywhere, whether in an office or at home or on mobile or on a VPN. Docker Desktop is designed to ensure that networking “just works” for all of these use-cases in all of these scenarios. This post describes the tools and techniques we use to make this happen, starting with everyone’s favorite protocol suite: TCP/IP.

TCP/IP

When containers want to connect to the outside world, they will use TCP/IP. Since Linux containers require a Linux kernel, Docker Desktop includes a helper Linux VM. Traffic from containers therefore originates from the Linux VM rather than the host, which causes a serious problem.

Many IT departments create VPN policies which say something like, “only forward traffic which originates from the host over the VPN”. The intention is to prevent the host accidentally acting as a router, forwarding insecure traffic from the Internet onto secure corporate networks. Therefore if the VPN software sees traffic from the Linux VM, it will not be routed via the VPN, preventing containers from accessing resources such as internal registries.

Docker Desktop avoids this problem by forwarding all traffic at user-level via vpnkit, a TCP/IP stack written in OCaml on top of the network protocol libraries of the MirageOS Unikernel project. The following diagram shows the flow of packets from the helper VM, through vpnkit and to the Internet:

When the VM boots it requests an address using DHCP. The ethernet frame containing the request is transmitted from the VM to the host over shared memory, either through a virtio device on Mac or through a “hypervisor socket” (AF_VSOCK) on Windows. Vpnkit contains a virtual ethernet switch (mirage-vnetif) which forwards the request to the DHCP (mirage/charrua) server.

Once the VM receives the DHCP response containing the VM’s IP address and the IP of the gateway, it sends an ARP request to discover the ethernet address of the gateway (mirage/arp). Once it has received the ARP response it is ready to send a packet to the Internet.

When vpnkit sees an outgoing packet with a new destination IP address, it creates a virtual TCP/IP stack to represent the remote machine (mirage/mirage-tcpip). This stack acts as the peer of the one in Linux, accepting connections and exchanging packets. When a container calls connect() to establish a TCP connection, Linux sends a TCP packet with the SYNchronize flag set. Vpnkit observes the SYNchronize flag and calls connect() itself from the host. If the connect() succeeds, vpnkit replies to Linux with a TCP SYNchronize packet which completes the TCP handshake. In Linux the connect() succeeds and data is proxied in both directions (mirage/mirage-flow). If the connect() is rejected, vpnkit replies with a TCP RST (reset) packet which causes the connect() inside Linux to return an error. UDP and ICMP are handled similarly.

In addition to low-level TCP/IP, vpnkit has a number of built-in high-level network services, such as a DNS server (mirage/ocaml-dns) and HTTP proxy (mirage/cohttp). These services can be addressed directly via a virtual IP address / DNS name, or indirectly by matching on outgoing traffic and redirecting dynamically, depending on the configuration.

TCP/IP addresses are difficult to work with directly. The next section describes how Docker Desktop uses the Domain Name System (DNS) to give human-readable names to network services.

DNS

Inside Docker Desktop there are multiple DNS servers:

DNS requests from containers are first processed by a server inside dockerd, which recognises the names of other containers on the same internal network. This allows containers to easily talk to each other without knowing their internal IP addresses. For example in the diagram there are 3 containers: “nginx”, “golang” and “postgres”, taken from the docker/awesome-compose example. Each time the application is started, the internal IP addresses might be different, but containers can still easily connect to each other by human-readable name thanks to the internal DNS server inside dockerd.

All other name lookups are sent to CoreDNS (from the CNCF). Requests are then forwarded to one of two different DNS servers on the host, depending on the domain name. The domain docker.internal is special and includes the DNS name host.docker.internal which resolves to a valid IP address for the current host. Although we prefer if everything is fully containerized, sometimes it makes sense to run part of an application as a plain old host service. The special name host.docker.internal allows containers to contact these host services in a portable way, without worrying about hardcoding IP addresses.

The second DNS server on the host handles all other requests by resolving them via standard OS system libraries. This ensures that, if a name resolves correctly in the developer’s web-browser, it will also resolve correctly in the developer’s containers. This is particularly important in sophisticated setups, such as pictured in the diagram where some requests are sent over a corporate VPN (e.g. internal.registry.mycompany) while other requests are sent to the regular Internet (e.g. docker.com).

Now that we’ve described DNS, let’s talk about HTTP.

HTTP(S) proxies

Some organizations block direct Internet access and require all traffic to be sent via HTTP proxies for filtering and logging. This affects pulling images during build as well as outgoing network traffic generated by containers.

The simplest method of using an HTTP proxy is to explicitly point the Docker engine at the proxy via environment variables. This has the disadvantage that if the proxy needs to be changed, the Docker engine process must be restarted to update the variables, causing a noticeable glitch. Docker Desktop avoids this by running a custom HTTP proxy inside vpnkit which forwards to the upstream proxy. When the upstream proxy changes, the internal proxy dynamically reconfigures which avoids having to restart the Docker engine.

On Mac Docker Desktop monitors the proxy settings stored in system preferences. When the computer switches network (e.g. between WiFi networks or onto cellular), Docker Desktop automatically updates the internal HTTP proxy so everything continues to work without the developer having to take any action.

This just about covers containers talking to each other and to the Internet. How do developers talk to the containers?

Port forwarding

When developing applications, it’s useful to be able to expose UIs and APIs on host ports, accessible by debug tools such as web-browsers. Since Docker Desktop runs Linux containers inside a Linux VM, there is a disconnect: the ports are open in the VM but the tools are running on the host. We need something to forward connections from the host into the VM.

Consider debugging a web-application: the developer types docker run -p 80:80 to request that the container’s port 80 is exposed on the host’s port 80 to make it accessible via http://localhost. The Docker API call is written to /var/run/docker.sock on the host as normal. When Docker Desktop is running Linux containers, the Docker engine (dockerd in the diagram above) is a Linux program running inside the helper Linux VM, not natively on the host. Therefore Docker Desktop includes a Docker API proxy which forwards requests from the host to the VM. For security and reliability, the requests are not forwarded directly over TCP over the network. Instead Docker Desktop forwards Unix domain socket connections over a secure low-level transport such as shared-memory hypervisor sockets via processes labeled vpnkit-bridge in the diagram above.

The Docker API proxy can do more than simply forward requests back and forth. It can also decode and transform requests and responses, to improve the developer’s experience. When a developer exposes a port with docker run -p 80:80, the Docker API proxy decodes the request and uses an internal API to request a port forward via the com.docker.backend process. If something on the host is already listening on that port, a human-readable error message is returned to the developer. If the port is free, the com.docker.backend process starts accepting connections and forwarding them to the container via the process vpnkit-forwarder, running on top of vpnkit-bridge.

Docker Desktop does not run with “root” or “Administrator” on the host. A developer can use docker run –privileged to become root inside the helper VM but the hypervisor ensures the host remains completely protected at all times. This is great for security but it causes a usability problem on macOS: how can a developer expose port 80 (docker run -p 80:80) when this is considered a “privileged port” on Unix i.e. a port number < 1024? The solution is that Docker Desktop includes a tiny helper privileged service which does run as root from launchd and which exposes a “please bind this port” API. This raises the question: “is it safe to allow a non-root user to bind privileged ports?”

Originally the notion of a privileged port comes from a time when ports were used to authenticate services: it was safe to assume you were talking to the host’s HTTP daemon because it had bound to port 80, which requires root, so the admin must have arranged it. The modern way to authenticate a service is via TLS certificates and ssh fingerprints, so as long as system services have bound their ports before Docker Desktop has started – macOS arranges this by binding ports on boot with launchd –  there can be no confusion or denial of service. Accordingly, modern macOS has made binding privileged ports on all IPs (0.0.0.0 or INADDR_ANY) an unprivileged operation. There is only one case where Docker Desktop still needs to use the privileged helper to bind ports: when a specific IP is requested (e.g. docker run -p 127.0.0.1:80:80), which still requires root on macOS.

Summary

Applications need reliable network connections for lots of everyday activities including: pulling Docker images, installing Linux packages, communicating with database backends, exposing APIs and UIs and much more. Docker Desktop runs in many different network environments: in the office, at home and while traveling on unreliable wifi. Some machines have restrictive firewall policies installed. Other machines have sophisticated VPN configurations. For all these use-cases in all these environments, Docker Desktop aims to “just work”, so the developer can focus on building and testing their application (rather than debugging ours!)

If building this kind of tooling sounds interesting, come and make Docker Desktop networking even better, we are hiring see https://www.docker.com/career-openings 

DockerCon2022

Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post How Docker Desktop Networking Works Under the Hood appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

How Docker containers are supporting the COVID-19 genomic monitoring effort

This is a guest blog post from Dr. Kelsey Florek and Curtis Kapsak. A video presentation of this post from Docker’s Community All-Hands can be found here.

Dr. Kelsey Florek is the Senior Genomics and Data Scientist at the Wisconsin State Laboratory of Hygiene and a steering committee member of the State Public Health Bioinformatics group (StaPH-B).

Curtis Kapsak is a Bioinformatics Scientist at Theiagen Genomics and is a member of the State Public Health Bioinformatics group (StaPH-B). 

The rapid appearance and global spread of a novel Severe Acute Respiratory Syndrome (SARS) virus in 2019 pushed public health laboratories to develop new methods for genomic monitoring efforts on a scale never seen before. Adding to this challenge, the approaches typically used in genomic data analysis often rely on cutting edge and often niche open source software and libraries that increase the complexity of setting up analytical pipelines or workflows. This along with a varying landscape of compute environments ranging from on-prem workstations to public Cloud created a significant barrier for many laboratories attempting to perform viral genomic monitoring.

Developing a library of dockerized tools

Public health laboratories inherently need to meet rigorous quality control and quality assurance standards. The tests performed in public health laboratories are either reported back to clinics to be used for patient care or used in aggregate to inform public health interventions or outbreak investigations. Analytical workflows are held to the same standards as other laboratory developed tests and to support this effort, the State Public Health Bioinformatics consortium (StaPH-B) started developing a repository of dockerized software that was commonly used in public health genomic data analyses, StaPH-B/docker-builds. This repository was designed to address the need for accessible software that is both highly reliable and reproducible. Combined with a usage guide, this repository provided a centralized location of maintained and tested open source tools to support laboratories developing analysis workflows.

Since its initial development in 2018 the StaPH-B/docker-builds repository has grown to contain multiple versions of over 90 different analytical tools from 19 different contributors, with several of the COVID-19 specific images achieving over 1 million pulls. Between March 2021 and January 2022 as more laboratories began genomic monitoring we saw a logarithmic increase in the number of Docker image pulls on core COVID-19 genomic analysis software.

Supporting COVID-19 genomic analytics

Bioinformatic pipelines or workflows consist of a variety of tools and often start from a form of raw or primary DNA sequencing data. These tools perform a variety of transformative or summary tasks and vary in both their computational requirements and dependencies. The process of sequencing the SARS-CoV-2 viral genome involves sectioning off the viral genome and sequencing small portions of the DNA in parallel. The result is a dataset containing hundreds of thousands to millions of short strings containing A’s, T’s, C’s, and G’s in a variety of sequence combinations. COVID-19 workflows then take these datasets, reconstruct the genome and use a variety of techniques to then characterize the virus.

Many laboratories across the globe have moved towards using a dedicated workflow language like WDL or Nextflow for their analytical workflows. Combining a workflow language with dockerized software allows for the creation and routine usage of workflows that are highly portable and easily adapted to a variety of compute environments. This gives laboratories the ability to run small datasets on a laptop or scale to a high performance compute cluster or cloud environment for large datasets. Additionally, these workflow approaches allow developing a modular analysis framework that enables swapping out software as new versions are released or issues are identified. With the rapid and constant evolution of the virus that causes COVID-19, updates to classification software are also frequently updating to maintain the ability to accurately identify variants.

Keeping up to speed with viral evolution

The COVID-19 virus evolves a bit slower than influenza accruing on average two mutations per month and different variants (Alpha, Delta, Omicron, etc.) are differentiated by various combinations of mutations. Classifying a virus requires constructing a phylogenetic tree that models the relationship of the new virus to other viruses. However, constructing a tree to compare each new virus to every previous virus is both computationally expensive and impractical. To address this, two commonly used methods have emerged including using a set of selected reference viruses to build a tree (Nextclade) or machine learning to classify mutational patterns (Pangolin). Both of these approaches require regular updates to ensure classification is occurring accurately with the most recent information. Leveraging containerization, StaPH-B has been able to maintain images with the latest models allowing users to run workflows knowing they are using the most up to date, robust, and tested classification tools.

Summary

The highly portable, scaleable, and efficient nature of containerization has transformed how public health disease monitoring is performed. The implementation of containerized workflows have enabled laboratories to quickly adopt complex analytical workflows, which in turn has grown the scale of the viral monitoring effort. The open source repository maintained by StaPH-B would not be possible without the community of bioinformaticians driving innovation. With more laboratories turning to sequencing and complex analytics there is a growing demand for people to bridge the gap between biology and informatics. If you are interested in a career in bioinformatics and using data to solve health problems be sure to check out the APHL-CDC Bioinformatics Fellowship!

DockerCon2022

Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post How Docker containers are supporting the COVID-19 genomic monitoring effort appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

File Sharing with Docker Desktop

One of the topics users of Docker Desktop often ask us about is file sharing. How do I see my source code inside my container? What’s the difference between a volume and a bind mount? Why is file sharing slower than on Linux, and how can I speed it up? In this blog post, I’ll cover the options you have, some tips and tricks, and finish with a sneak preview of what we’re currently working on.

Bind mounts and volumes

Suppose you run an Ubuntu container with docker run -it ubuntu bash. You’ll quickly find that (1) the container has its own filesystem, based on the filesystem in the Ubuntu image; (2) you can create, delete and modify files, but your changes are local to the container and are lost when the container is deleted; (3) the container doesn’t have access to any other files on your host computer.

So the natural next questions are, how can I see my other files? And how can my container write data that I can read later and maybe use in other containers? This is where bind mounts and volumes come in. Both of these use the -v flag to docker run to specify some files to share with the container.

The first option most people encounter is the bind mount, where part of your local filesystem is shared with the container. For example, if you run

docker run -it -v /users/stephen:/my_files ubuntu bash

then the files at /users/stephen will be available at /my_files in the container, and you can read and write to them there. This is very simple and convenient, but if you’re using Docker Desktop a named volume may have better performance, for reasons I’ll explain in the next section.

The second option, a named volume, is a filesystem managed by Docker. You can create a named volume with a command like docker volume create new_vol, and then share it into the container using the -v flag again:

docker run -it -v new_vol:/my_files ubuntu bash

These volumes persist even after the container has been deleted, and can be shared with other containers. You can also browse their contents from the Docker Desktop UI, using the Volumes tab that we added recently (and that is now free for all users including Docker Personal).

Performance considerations

To understand the performance differences between the options, we first have to talk briefly about how Docker Desktop works. Many people imagine that Docker Desktop is just a UI on top of some open source tools, but that’s only a small part of what it is. Docker Desktop is fundamentally an environment to develop and run Linux containers on your Mac or Windows machine, with seamless integration into the host so that it appears as if they’re running natively. It does this by setting up a Linux VM (or optionally a WSL 2 environment on Windows) in which to run the Docker engine and your containers, and then passing CLI commands, networking and files between the host and the VM.

Unfortunately it’s in the nature of virtualization that there is always an unavoidable small overhead in crossing the host-VM boundary. It’s only tiny, but in a development environment with a huge source tree and lots of reads and writes, it adds up, and can visibly affect performance. And because Docker Desktop does such a good job of hiding the underlying VM, it’s not obvious why this is happening. On Linux, the container has direct access to the bind-mounted filesystem, and because the implementation on Mac and Windows “feels native”, people intuitively expect the same performance there.

Named volumes do not suffer from the same problem because they are created inside the VM’s own filesystem, so they are as fast as on a Linux machine. In WSL 2, Windows manages file sharing rather than Docker managing it, but the same performance considerations apply: files mounted from the Windows file system can be slow, named volumes are fast, but in this case there is another option: files stored in the Linux filesystem are also inside the WSL VM so are fast too.

Best practices and tips

This gives us the main tip in optimizing performance. It’s convenient to use bind mounts at first, and you may find that they are fine for your use case. But if performance becomes a problem, then (1) make sure that you’re only sharing what you need to share, and (2) consider what could be shared in some other way than a bind mount. You have several options for keeping files inside the VM, including a named volume, Linux files in WSL, and the container’s own filesystem: which to use will depend on the use case. For example:

Source code that you are actively editing is an appropriate use of a bind mountLarge, static dependency trees or libraries could be moved into a named volume, or WSL, or even baked into the container imageDatabases are more appropriate in a named volume or WSLCache directories and logfiles should be put in a named volume or WSL (if you need to keep them after the container has stopped) or in the container’s own filesystem (if they can disappear when the container does).Files that the container doesn’t need shouldn’t be shared at all. In particular, don’t share the whole of your home directory. We have seen some people do this habitually so that they’ll always have access to whatever files they need, but unlike on Linux it’s not “free”.

One remaining option if you really need a bind mount for some large or high-traffic directory is a third-party caching/syncing solution, for example Mutagen or docker-sync. These essentially copy your files inside the VM for faster read/write access, and handle syncing (in one or both directions) between the copy and the host. But it involves an extra component to manage, so named volumes are still preferred where possible.

The future

We have used a variety of file sharing implementations over the years (Samba and gRPC FUSE on Windows Hyper-V; osxfs and gRPC FUSE on Mac; and Windows uses 9P on WSL 2). We have made some performance improvements over time, but none of them have been able to match native performance. But we are currently experimenting with a very promising new file sharing implementation based on virtiofs. Virtiofs is new technology that is specifically designed for sharing files between a host and a VM. It is able to make substantial performance gains by using the fact that the VM and the host are running on the same machine, not across a network. In our experiments we have seen some very promising results.

We have already released a preview of this technology for Docker Desktop for Mac, which you can get from our public roadmap (it requires macOS 12.2), and we are also planning to use it for the forthcoming Docker Desktop for Linux. We think that it will be able to make bind mounts a lot faster (although we would still recommend named volumes or the container’s own filesystem for appropriate use cases). We would love to hear your experiences of it.

Next steps

If you want to go into more depth about these topics, I recommend a talk that one of our Docker Captains, Jacob Howard, gave at DockerCon 2021 entitled A Pragmatic Tour of Docker Filesystems. It’s got loads of great information and practical advice packed into only 26 minutes!

To follow the progress of our current work on virtiofs, subscribe to the ticket on our public roadmap. That’s where we post preview builds, and we’d love you to try them and give us your feedback on them.

DockerCon2022

Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post File Sharing with Docker Desktop appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Docker Captain Take 5 – Thomas Shaw

Docker Captains are select members of the community that are both experts in their field and are passionate about sharing their Docker knowledge with others. “Docker Captains Take 5” is a regular blog series where we get a closer look at our Captains and ask them the same broad set of questions ranging from what their best Docker tip is to whether they prefer cats or dogs (personally, we like whales and turtles over here). Today, we’re interviewing Thomas Shaw, one of our Docker Captains. He works as a Principal Automation Engineer at DIGIT Game Studios and is based in Ireland.

How/when did you first discover Docker?

I remember it like it was yesterday.  he date was August 23rd 2013 and I was working as a Build Engineer for Demonware.  During a visit to the Vancouver office, one of the developers named Mohan Raj Rajamanickham mentioned “a cool new tool” called Docker. It sounded too good to be true but I downloaded version 0.5 the next day while waiting for the flight back home to Dublin, Ireland. I played with the Docker CLI for several hours on the flight and that was it.  Before the plane had touched the runway in Dublin I was completely sold on the potential of Docker. It solved an immediate problem faced by developers and build engineers alike, dependency hell.

Over the following 12 months we replaced our bare metal build agents with containerized build agents. It was a primitive approach at first. We built a CentOS 5 VM, tarred it up and created a container image from it. This was the base image on which we ran builds and tests over the next 2 years. We went from 8 bare metal build agents, each unique, each manually setup, each with different tooling versions to 4 build agents with just Docker installed.

It was a simple approach but it eliminated the management of several unique build agents. We saw a number of other benefits too, such as better build stability, increased build throughput by 300% and most importantly teams now owned their own dependencies. This approach worked well and around 2015 we started looking at moving our CI/CD pipelines into AWS. We originally took a hybrid approach and ran the majority of builds and tests in our own datacenter and just a handful in AWS. This was easier than expected. Docker made our workloads portable and we began to leverage the scalability of AWS for running tests. The unit tests (which were actually functional tests) for one of our main projects was taking over 1 hour per commit. Using containers we were able to split the tests across multiple containers on multiple build agents and we reduced the execution time to around 10 minutes. It was at this point that more folks started to pay attention to the potential of Docker.

What is your favorite Docker command?

I really enjoy “docker diff”. I find it incredibly useful to see what files/directories are being added or modified by the process within the container. It’s great for debugging. A close second would be “docker stats”.

What is your top tip for working with Docker that others may not know?

When possible, own your own base image and pin tooling versions. It’s convenient to use the public images on Docker Hub but when working in an organization where hundreds of developers are relying on the same base image try to bring it in house. We set up an image “bakery” at Demonware where we would build our base images nightly, with pinned versions of tools included, ran extensive tests, triggered downstream pipelines and verified that our base image was always in a good state. From experience it’s the base image where the most “bloat” occurs and keeping it lean can also help when moving the image around your infrastructure.

What’s the coolest Docker demo you have done/seen ?

My favorite demo was by Casey Bisson from Joyent. In the demo he showed off how Triton worked and how he could scale from a single container running locally out to an entire datacenter by just updating a single endpoint. This was in 2016 and I still love the simplicity of this approach.

What have you worked on in the past six months that you’re particularly proud of?

I’ve been using containers daily since late 2013 and still find new uses for them regularly. I’m currently working on improving developer tooling and UX at DIGIT Games Studios in Dublin. Part of this work includes containerizing our tooling and making it accessible in multiple ways including from Slack, command line, callable by API and even runnable from a spreadsheet. Docker enables us to bring the tooling closer to the end user whether they are technical or non technical. Rolling out updates to tools is now trivial.

What do you anticipate will be Docker’s biggest announcement this year?

Development Environments (DE) has received a huge amount of positive feedback based on the early announcements. I think the potential of DE is huge. To have these environments tightly integrated into a developers IDE, easily shareable and customizable will remove existing friction and help developers move from idea to production with greater velocity.

What are some personal goals for the next year with respect to the Docker community?

Our last Docker Dublin Meetup was in February 2020 and with a community of over 3000 members I’d like to start the in-person meetups again in 2022. I’d also like to continue running more Docker Workshops around Ireland and take the power of containerization out to other communities.

What talk would you most love to see at DockerCon 2022?

Any talks that include gaming would be great. I particularly loved the live migration demo of Quake a few years ago. Some studios are doing really cool stuff with containers. As early adopters of containers, Demonware may have some useful experiences to share with regard to their journey from self-hosted Kubernetes to the cloud. 

Looking to the distant future, what is the technology that you’re most excited about and that you think holds a lot of promise?

That’s a tough question. Any technology that focuses on the user experience first and is a joy to use excites me. From a gaming perspective Augmented Reality has so much potential, particularly for mobile gaming. I’m not going to mention Blockchain or NFTs since I don’t know enough about either to comment. They may have their place but if they negatively impact the environment as suggested then perhaps they need to go back to the drawing board for a few more iterations.

Rapid fire questions…

What new skill have you mastered during the pandemic?

Mindfulness.

Cats or Dogs?

Dogs.

Salty, sour or sweet?

Sweet.

Beach or mountains?

Mountains.

Your most often used emoji?

DockerCon2022

Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post Docker Captain Take 5 – Thomas Shaw appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

How to Purchase a Docker Subscription from a Reseller

With the grace period for the new Docker subscription service agreement ending very soon on January 31, 2022, we want to make it easy for customers to use their preferred reseller to purchase a Docker subscription.

That’s why we recently announced that your preferred reseller can now purchase a Docker Business subscription through Nuaware. That’s right, Docker and Nuaware have teamed up to advance the adoption of Docker throughout the software engineering departments of large organizations. Nuaware has established relationships with 1,000’s of resellers around the globe.

Nuaware is a Trusted Partner

Nuaware is a specialized distributor in developer tools, DevOps, Cloud, and Cloud-Native technologies. They help large enterprises adopt modern architectures by supporting them with the right products, training, and partner ecosystem. Nuaware is part of Exclusive Networks, a global provider of cybersecurity and cloud solutions with offices in over 50 countries across 5 continents with a reseller network exceeding 18,000 partners. Nuaware was founded with the view that new platform and software development technologies are an ecosystem business and in order for an enterprise to successfully adopt new technologies like microservice architectures in production, they need specialist partners in many areas. Nuaware helps select and introduce the right technologies, partners, and training to take ideas into production.

Serving Our Customers Around the Globe

Developers using Docker are more productive, build more secure software, and are able to collaborate more effectively. Combined with Nuaware’s developer tool expertise and the reach of Exclusive Networks’ channel ecosystem of 18,000 specialist channel partners, Docker has an unparalleled ability to service our clients’ needs around the globe. No matter how you prefer or need to purchase your software, Docker has you covered. 

Get Started Today

Learn more about Docker Business and get started today by visiting https://www.nuaware.com/docker
The post How to Purchase a Docker Subscription from a Reseller appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Seamless Sign-in with Docker Desktop 4.4.2

Starting with Docker Desktop 4.4.2 we’re excited to introduce a new authentication flow that will take you through the browser to sign in, simplifying the experience and allowing users to get all the benefits of autofill from whatever browser password manager they may use. Gone are the days of going to your browser, opening your password manager, finding your Docker password, copying, and then returning back to the dashboard or command line to paste it in (maybe I’m alone here, but I can never remember my passwords on my own!). 

As part of this authentication change, we’ve also introduced Single Sign-on for users with a business subscription that enables benefits such as automated and secure onboarding of developers into the Docker platform, easy user management for managers and admins, and seamless authentication for Docker users. Read our blog to learn more about SSO.

Getting Started with the new Authentication Flow 

On versions 4.4.2 and above, when users click on “Sign in” from the Docker Dashboard or the Docker Menu, they will be redirected to their default browser.

Once in the browser, you will be prompted to enter your Docker ID. This will take you through the appropriate flow. If your organization has Single Sign-on enabled, you will get redirected to your Identity Provider. 

For the standard authentication pattern, users will then be sent to a screen to fill in their username and password. Then click “continue”.

Once you have successfully signed in you will get asked if you want to be taken back to the Docker Dashboard. You can choose to always allow opening Docker to reduce the number of steps in the future.

You’ll be returned to the Docker Dashboard, ready to continue working! 

It’s important to note that the command line interface (CLI) login flow and logging into Docker in other tools, such as Visual Studio Code, will continue to work as they do today. You can choose to use a personal access token (PAT) for these flows. 

Just update to or download 4.4.2+ to get all the benefits of a simplified sign in experience.

Introduced a fix for CVE-2021-45449 

Docker Desktop version 4.3.0 and 4.3.1 have a bug that may log sensitive information (access token or password) on the user’s machine during login. This only affects users if they are on Docker Desktop 4.3.0, 4.3.1 and the user has logged in while on 4.3.0, 4.3.1. Gaining access to this data would require having access to the user’s local files. Additionally, these logs may be included when users upload diagnostics, meaning access tokens and passwords might have been shared with Docker. We have deleted all potentially sensitive diagnostic files from our data storage and will continue to delete diagnostics reported from the affected versions on an ongoing basis. This vulnerability has been fixed in version 4.3.2 or higher. Learn more on our security page.

Chat with us!

We’d love to get your input on our work through user interview sessions. During these live feedback sessions we will often show work-in-progress features or ideas to get your input on, or ask you more about how you use Docker to discover any pain points when working with the product. We use these insights to help prioritize our roadmap and improve the user experience. Each session is typically 30 minutes to an hour. If you would like to take part in one of our user research studies please sign up and we will reach out to you when we have something to research or test.

We’re also working on two of your highest voted items: improving Mac filesystem performance, and implementing Docker Desktop for Linux. For filesystem performance, we’ve been releasing experimental builds using VirtioFS, and we’d love your input on how it works for you: check out the roadmap item for the latest build. For Docker Desktop for Linux, we will have some experimental builds soon, so be sure to follow that roadmap item for news.

 What else would you like us to focus on? Have your say by adding a thumbs-up emoji to your highest priorities on the roadmap, or create a new roadmap ticket if your idea isn’t yet there. We can’t wait to hear from you.

DockerCon Live 2022  

Join us for DockerCon Live 2022 on Tuesday, May 10. DockerCon Live is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon Live 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post Seamless Sign-in with Docker Desktop 4.4.2 appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Introducing SSO for Docker Business

Single Sign-on (SSO) for Docker is now live! By enabling SSO, large organizations and enterprises can easily automate the onboarding and management of Docker users at scale. Users can authenticate using their organization’s standard identity provider (IdP). SSO is one of our most widely requested features, so we’re excited to ship this to our Docker Business customers.

Want to enable SSO for your organization? Here are the top things you need to know.

With SSO enabled, users can authenticate using their organization’s standard IdP.

How does SSO work in Docker?

SSO allows users to authenticate to Docker Hub and Docker Desktop using their organization’s standard identity provider (IdP) to access Docker. This will not only make it easier for new users to quickly get started with Docker using their organization-provided email, but it will also help large organizations scale their use of Docker in a more manageable and secure way. Docker currently supports SAML 2.0 and Azure Active Directory IdPs for easy implementation. Once SSO is enabled and configured for your organization, users must sign in to Docker Hub or Docker Desktop to initiate the SSO authentication process. 

How is SSO enabled?

SSO is available to organizations with an active Docker Business subscription. This means that customers under the other subscription tiers (e.g., Team) must first upgrade to a Docker Business account. Click here to learn how to upgrade your subscription. Customers with a Docker Business subscription, can visit our documentation for additional information on the enablement process.

Note: When SSO is enabled, logging into Docker via partner products (e.g., VS Code, Jfrog, etc.) will require Personal Access Tokens (PATs). 

How are users managed?

Users are managed through organizations in Docker Hub. To configure SSO, each user must already have an existing account in their organization’s IdP. When a user signs into Docker for the first time using their domain email address, they will be automatically added to the organization after the authentication is successful. All users must authenticate using the email domain specified during SSO setup (i.e., company email address). Admins can continue inviting new users to the organization using the Invite Member option in Docker Hub. 

How do we convert existing Docker users from non-SSO to SSO?

To convert existing Docker users from a non-SSO account to SSO, admins must verify:

Users have a company email address and account in the IdPUsers have the latest version of Docker Desktop (currently version 4.4.2) installed on their machinesUsers have created a Personal Access Token (PAT) to replace their passwords to allow them to log in through the Docker CLIAll CI/CD pipeline automation systems have replaced their passwords with PATsUsers with email addresses that include the “+” symbol are either added to your IdP or otherwise updated to not include the “+” symbol.

For additional requirements, please refer to our documentation.

What impact can be expected when onboarding users to SSO?

SSO can be enforced for your users once the steps (summarized above and in our documentation) are completed. After SSO is enforced, users can begin signing in using their organization-provided email and password, and then it’s business as usual. Please note that for users logging into Docker directly from the Docker CLI or via partner products (e.g., VS Code, Jfrog, etc.), Personal Access Tokens (PATs) may be required. 

For more guidance on how to roll out SSO within your organization, visit our documentation for more information.

Consider making the move today for access to SSO for Docker and other premier features for management and security at scale. Download our latest whitepaper and watch our webinar on-demand to learn more. You can also visit our public roadmap where you can leave feedback on what you want to see next for user management.

DockerCon Live 2022  

Join us for DockerCon Live 2022 on Tuesday, May 10. DockerCon Live is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon Live 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post Introducing SSO for Docker Business appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Guest Blog: Deciding Between Docker Desktop and a DIY Solution

Guest author Ben Hall is the lead technical developer for C# .NET at gov.uk (a United Kingdom public sector information website) and a .NET Foundation foundation member. He worked for nine years as a school teacher, covering programming and computer science. Ben enjoys making complex topics accessible and practical for busy developers.

Deciding Between Docker Desktop and a DIY Solution

At the heart of the Docker experience is Docker Engine. Docker Desktop’s ready-to-use solution for building containerized applications includes Docker Engine and all the other tooling and setup you need to start developing right away.

Developers can create a “DIY” Docker implementation around Docker Engine manually. Some organizations may prefer the flexibility and control of doing it themselves. But opting for a DIY Docker Engine solution requires much more engineering, development, and setup. Docker and its Windows companion, WSL, are relatively complex, so the DIY approach isn’t for everyone.

In this article, we’ll help you decide which approach is right for you and your organization. To illustrate, we’ll draw comparisons between what Docker Desktop offers and a DIY Docker setup on Windows.

Setting Up Docker on Windows

This article on failingfast.io describes the main steps for a manual installation on Windows, which involves creating a WSL 2 distro, setting up a Docker repository, and installing the Docker Engine on the WSL 2 distro additional setup. This process is a bit fragile, so prepare for troubleshooting before you’re up and running. And it’s only a guide to getting started. Most use cases will need further setup, including:

Configuring Docker to start on bootLoggingAccepting connections to Docker daemon from remote hostsConfiguring remote accessFixing IP forwarding problems

Setting up Docker Desktop is a very different experience. You simply download and run the latest Docker Desktop installer — it automatically completes all the work. We’re up and running in a few minutes, ready to deploy containers.

Cutting Edge and Stable

Docker Desktop and the DIY implementation that we linked to share a common foundation on Windows Subsystem for Linux (WSL) 2 that enables developers to run a Linux environment directly on Windows.

WSL 2 significantly improved memory use, code execution, and compatibility. It achieved this through an architectural shift to a full Linux kernel, which supports Linux containers running natively, without emulation.

Working closely with Microsoft and Windows Insider, Docker was quick to adopt this beneficial emerging technology as the primary backend for Docker Desktop. Docker then released a technical preview far in advance of WSL 2 reaching general availability in Windows. Every effort was made also, to maintain feature parity with the previous version that used Hyper-V.

We can add Docker Desktop to our developer tooling, confident that it will continue to support that latest technology while avoiding breaking changes to the experience we are accustomed to.

Software Updates

Docker Desktop manages everything, from setup through to future kernel patches. And because it’s a complete bundle, the automatic software updates will keep all the tools installed on it up-to-date and secure, including the Docker Engine itself. That’s one less machine image to manage in-house!

With a DIY Docker setup, it’s up to you to keep up with all security patches and other updates. A DIY solution will also provide you with plenty of ongoing problems that need solving. So, be sure to multiply those developer hours across a large organization when you are calculating the ROI for Docker Desktop.

Networking

Docker Desktop will automatically propagate configured HTTP/HTTPS proxy settings to Docker to use when pulling containers.

It will also function properly when attached to a VPN. It achieves this by intercepting traffic from containers and injecting it into Windows as if it originated from the Docker application itself.

Pause and Resume

This feature was requested by a user on the public roadmap for Docker Desktop. It’s not the biggest feature ever, but it’s another great reminder that Docker Desktop is under active development. It’s continually being improved in response to user feedback, implemented with monthly releases.

Users can now pause a Docker Desktop session to reduce CPU usage and conserve battery life. When paused, the current state of all your containers is saved in memory and all processes are frozen.

Volume Management

Volumes are the standard approach to persisting any data that Docker containers work with, including files shared between containers. Unlike bind mounts, which work directly with host machine files, volumes are managed by Docker, offering several advantages.

You’ll face two big challenges when working with Docker volumes manually in the Docker CLI:

It can be difficult to identify which container each volume belongs to, so clearing up old volumes can be a slow process.Transferring content in and out of volumes is more convoluted than it really needs to be.

Docker Desktop provides a solution for this by providing a view in the Dashboard to explore volumes. In this view, you can:

Easily identify which volumes are being used See which containers are using a volumeCreate and delete volumesExplore the files and folders in a volume, including file sizesDownload files from volumesSearch and sort by name, date, and size

Kubernetes Integration

Although there are too many features to explore in a single article, we should take a look at the Kubernetes integration in Docker Desktop.

Kubernetes has become a standard for container orchestration, with 83 percent of respondents to the 2020 CNCF Survey reporting that they use it in production.

Granted, we don’t need Kubernetes to get Docker’s benefits in local development, like the isolation from the host system. Plus, we can even use Docker Compose 2.0 to run multiple containers with some nifty networking features. But if you’re working on a project that will deploy to Kubernetes in production, using a similar environment locally is a wise choice.

In the past, a local Kubernetes instance was something else to set up, and the costs in developer time didn’t offer enough benefit to some. This is likely still the case for a DIY Docker solution.

Docker Desktop, in contrast, comes with a standalone Kubernetes server and client for local testing. It’s an uncomplicated, no-configuration, single-node cluster. You can switch to it through the UI, as the image below shows, or in the usual way with kubectl config use-context.

Native Apple Silicon Support

In 2021, a version of Docker Desktop for Mac that could fully leverage the latest M1 chip reached general availability. There are already over 145,000 ARM-based images on Docker Hub. This Apple Silicon version supports multi-platform images, which means you can build and run images for x86 and ARM architectures without complex cross-compilation environments.

This was very well-received because the emulation offered by Rosetta 2, which offers acceptable functionality for many common applications, isn’t sufficient to run containers.

Costs and Scalability

The DIY alternative requires a great deal of engineering time to build and configure, with an ongoing maintenance commitment for updating, patching, and troubleshooting the container environment. Each developer in an organization will carry out most of this work individually every time they work in a fresh environment. 

This approach doesn’t scale well! It means developers won’t be spending time on activities directly benefiting the business, like new features. None of us enjoy a sprint review where we have to explain that we didn’t deliver a feature because of problems with or work setting up development environments.

Containerization should help facilitate product delivery. What Docker Desktop sets out to achieve is not new. We have always invested in programming IDEs and other tooling that bundle functionality in a single, user-friendly package to improve productivity.

To help you determine whether Docker Desktop is right for your organization from a cost perspective, Jeremy Castile has some guidance to help you assess the ROI.

Working with Multiple Environments

Developers widely accept that build artifacts must be immutable — the same application, built, must move through QA to production. The next level, if you’d like, is packaging an application and its dependencies together. This helps to further maintain consistency between development, testing, and production environments.

We risk not realizing this benefit if the process is too complicated. Organizations have introduced many great tools and processes to teams, only for these tools to gather dust because the entry bar for the required skills is too high.

This situation is more prominent in QA teams. Many testers are technical, but more typically, they have a particular set of skills geared towards testing. Since QA is one group set to benefit a great deal from consistent testing environments, consider what they are most likely to use.

Introducing Dev Environments

To improve the experience further for these scenarios, Docker Desktop has added a new collaborative development feature, currently in preview, called Dev Environments.

Switching git branches or environments usually requires lots of manual changes to configuration, dependencies, and other environment setup before it’s possible to run the code.

The new feature makes it easy to keep the environment details themselves in source control with the code. With a click of a button, a developer can share their work-in-progress and its dependencies via Docker Hub. This means developers can easily switch to fully functioning instances of each other’s work to, for example, complete a pull request without having to change from their local branch and make all those environment changes.

Get started in Development Environments with the preview documentation.

Conclusion

Bret Fisher, an author who writes about Docker, summed up the need for Docker Desktop: “It’s really a testament to Docker Desktop that there isn’t a comparable tool for local Linux containers on macOS/Windows that solves 80% of what people typically need out of a container runtime locally.”

We’ve explored what Docker Desktop offers and along the way. We’ve also touched on the subjects of cost and ROI, setup and maintenance, scalability, and onboarding. Although some will prefer DIY Docker’s flexibility and control, Docker Desktop requires less setup effort and maintenance, offering a gentler learning curve for everyone from development to QA.

Perhaps the greatest challenge of a DIY solution is from a business value perspective. Developers love discovering how to do these things. So, a developer won’t necessarily track how many hours they spent over a week engaged in maintaining a DIY solution — the business will have no visibility into any productivity loss.If you’re still using a DIY solution for local development with Docker on Windows or macOS, learn more about Docker Desktop and download it to get started.
The post Guest Blog: Deciding Between Docker Desktop and a DIY Solution appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/