Kategorie: Kubernetes

How to Build and Test Your Docker Images in the Cloud with Docker Hub

Part 2 in the series on Using Docker Desktop and Docker Hub Together

Introduction

In part 1 of this series, we took a look at installing Docker Desktop, building images, configuring our builds to use build arguments, running our application in containers, and finally, we took a look at how Docker Compose helps in this process. 

In this article, we’ll walk through deploying our code to the cloud, how to use Docker Hub to build our images when we push to GitHub and how to use Docker Hub to automate running tests.

Docker Hub

Docker Hub is the easiest way to create, manage, and ship your team’s images to your cloud environments whether on-premises or into a public cloud.

This first thing you will want to do is create a Docker ID, if you do not already have one, and log in to Hub.

Creating Repositories

Once you’re logged in, let’s create a couple of repos where we will push our images to.

Click on “Repositories” in the main navigation bar and then click the “Create Repository” button at the top of the screen.

You should now see the “Create Repository” screen.

You can create repositories for your account or for an organization. Choose your Docker ID from the dropdown. This will create the repository for your Docker ID.

Now let’s give our repository a name and description. Type projectz-ui in the name field and a short description such as: This is our super awesome UI for the Projectz application. 

We also have the ability to make the repository Public or Private. Let’s keep the repository Public for now.

We can also connect your repository to a source control system. You have the option to choose GitHub or Bitbucket but we’ll be doing this later in the article. So, for now, do not connect to a source control system. 

Go ahead and click the “Create” button to create a new repository.

Your repository will be created and you will be taken to the General tab of your new repository.

This is the repository screen where we can manage tags, builds, collaborators, webhooks, and visibility settings.

Click on the Tags tab. As expected, we do not have any tags at this time because we have not pushed an image to our repository yet.

We also need a repository for your services application. Follow the previous steps and create a new repository for the projectz-services application. Use the following settings to do so:

Repository name: projectz-services

Description: This is our super awesome services for the Projectz application

Visibility: Public

Build Settings: None

Excellent. We now have two Docker Hub Repositories setup.

Structure Project

For simplicity in part 1 of this series, we only had one git repository. For this article, I refactored our project and broke them into two different git repositories to more align with today’s microservices world.

Pushing Images

Now let’s build our images and push them to the repos we created above.

Fork Repos

Open your favorite browser and navigate to the pmckeetx/projectz-ui repository.

Create a copy of the repo in your GitHub account by clicking the “Fork” button in the top right corner.

Repeat the processes for the pmckeetx/projectz-svc repository.

Clone the repositories

Open a terminal on your local development machine and navigate to wherever you work on your source code. Let’s create a directory where we will clone our repos and do all our work in.

$ cd ~/projects
$ mkdir projectz

Now let’s clone the two repositories you just forked above. Back in your browser click the green “Clone or download” button and copy the URL. Use these URLs to clone the repo to your local machine.

$ git clone https://github.com/[github-id]/projectz-ui.git ui
$ git clone https://github.com/[github-id]/projectz-svc.git services

(Remember to substitute your GitHub ID for [github-id] in the above commands)

If you have SSH keys set up for your github account, you can use the SSH URLs instead.

List local images

Let’s take a look at the list of Docker images we have locally on our machine. Run the following command to see a list of images.

$ docker images

You can see that I have the nginx, projectz-svc, projectz-ui, and node images on my machine. If you do not see the above images, that’s okay, we are going to recreate them now.

Remove local images

Let’s first remove projectz-svc and projectz-ui images. We’ll use the remove image (rmi) command. You can skip this step if you do not have the projectz-svc and projectz-ui on your local machine.

$ docker rmi projectz-svc projectz-ui

If you get the following or similar error: Error response from daemon: conflict: unable to remove repository reference “projectz-svc” (must force) – container 6b1b99cc899c is using its referenced image 6b9eadff19ae

This means that the image you are trying to remove is being used by a container and can not be removed. You need to stop and rm (remove) the container before you can remove the image. To do so, run the following commands.

First, find the running container:

$ docker ps -a

Here we can see that the container named services is using the image projectz-svc which we are trying to remove. 

Let’s stop and remove this container. We can do this at the same time by using the –force option to the rm command. 

If we tried to remove the container by using docker rm services without first stopping it, we would get the following error: Error response from daemon: You cannot remove a running container 6b1b99cc899c. Stop the container before attempting removal or force remove

So we’ll use the –force option to tell Docker to send a SIGKILL to the container and then remove it.

$ docker rm –force services

Do the same for the UI container, if it is still running.

Now that we stopped and removed the containers, we can now remove the images.

$ docker rmi projectz-svc projectz-ui

Let’s list our images again.

$ docker images

Now you should see that the projectz-ui and projectz-services images are gone.

Building images

Let’s build our images for the UI and Services projects now. Run the following commands:

$ cd [working dir]/projectz/services
$ docker build –tag projectz-svc .
$ cd ../ui
$ docker build –tag projectz-ui .

If you would like a more in-depth discussion around building images and Dockerfiles, refer back to part 1 of this series.

Pushing images

Okay, now that we have our images built, let’s take a look at pushing them to Docker Hub.

Tagging images

If you look back at the beginning of the post where we set up our Docker Hub repositories, you’ll see that we created the repositories in our Docker ID namespace. Before we can push our images to Hub, we’ll need to tag them using this namespace.

Open your favorite browser and navigate to Docker Hub and let’s review real quick.

Login to Hub, if you’ve not already done so, and take a look at the dashboard. You should see a list of images. Choose your Docker ID from the dropdown to only show images associated with your Docker ID.

Click on the row for the projectz-ui repository. 

Towards the top right of the window, you should see a docker command highlighting in grey.

This is the Docker Push command followed by the image name. You’ll see that this command uses your Docker ID followed by a slash followed by the image name and tag, separated by a colon. You can read more about pushing to repositories and tagging images in our documentation.

Let’s tag our local images to match the Docker Hub Repository. Run the following commands anywhere in your terminal.

$ docker tag projectz-ui [dockerid]/projectz-ui:latest
$ docker tag projectz-svc [dockerid]/projectz-svc:latest

(Remember to substitute your Docker ID for [dockerid] in the above commands)

Now list your local images and see the newly tagged images.

$ docker images

Pushing

Okay, now that we have our images tagged correctly, let’s push our images to Hub.

The first thing we need to do is make sure we logged into Docker Hub on the terminal. Although the repositories we created earlier are “public”, only the owner of the repository can push by default. If you would like to allow folks on your team to be able to push images and manage repositories. Take a look at Organizations and Teams in Hub.

$ docker login
Login with your Docker ID to push and pull images from Docker Hub…
Username:

Enter your username (Docker ID) and password.

Now we can push our images.

$ docker push [dockerid]/projectz-ui:latest
$ docker push [dockerid]/projectz-svc:latest

Open your favorite browser and navigate to Docker Hub, select one of the repositories we created earlier and then click the “Tags” tab. You will now see the images and tag we just pushed.

Automatically Build and Test Images

That was pretty straightforward but we had to run a lot of manual commands. What if we wanted to build an image, run tests and publish to a repository so we could deploy our latest changes?

We might be tempted to write a shell script and have everybody on the team run it after they completed a feature. But this wouldn’t be very efficient. 

What we want is a continuous integration (CI) pipeline. Docker Hub provides these features using AutoBuilds and AutoTests

Connecting Source Control

Docker Hub can be connected to GitHub and Bitbucket to listen to push notifications so it can trigger AutoBuilds.

I’ve already connected my Hub account to my GitHub account. To connect your own Hub account to your version control system follow these simple steps in our documentation.

Setup AutoBuilds

Let’s set up AutoBuilds for our two repositories. The steps are the same for both repositories so I’ll only walk you through one of them.

Open Hub in your browser, and navigate to the detail page for the projectz-ui repository.

Click on the “Builds” tab and then click the “Link to GitHub” button in the middle of the page.

Now in the Build Configuration screen. Select your organization and repository from the dropdowns. Once you select a repository, the screen will expand with more options.

Leave the AUTOTEST setting to Off and the REPOSITORY LINKS to Off also.

The next thing we can configure is Build Rules. Docker Hub automatically configures the first BUILD RULE using the master branch of our repo. But we can configure more.

We have a couple of options we can set for build rules. 

The first is Source Type which can either be a Branch or a Tag. 

Then we can set the Source, this is referring to either the Branch you want to watch or the Tag name you would like to watch. You can enter a string literal or a RegExp that will be used for matching.

Next, we’ll set the Docker Tag that we want to use when the image is built and tagged.

We can also tell Hub what Dockerfile to use and where the Build Context is located.

The next option turns off or on the Build Rule.

We also have the option to use the Build Cache.

Save and Build

We’ll leave the default Build Rule that Hub added for us. Click the “Save and Build” button.

Our Build options will be saved and an AutoBuild will be kicked off. You can watch this build run on the “Builds” tab of your image page.

To view the build logs, click on the build that is in progress and you will be taken to the build details page where you can view the logs.

Once the build is complete, you can view the newly created image by clicking on the “Tags” tab. There you will see that our image was built and tagged with “latest”.

Follow the same steps to set up the projectz-svc repository. 

Trigger a build from Git Push

Now that we see that our image is being built, let’s make a change to our project and trigger it from git push command.

Open the projectz-svc/src/routes.js file in your favorite editor and add the following code snippet anywhere before the module.exports = appRouter line at the bottom of the file.

…

appRouter.get( ‘/services/hello’, function( req, res ) {
res.json({ code: ‘success’, payload: ‘World’ })
})

…

module.exports = appRouter

Save the file and commit the changes locally.

$ git commit -am “add hello – world route”

Now, if we push the changes to GitHub, GitHub will trigger a webhook to Docker Hub which will in turn trigger a new build of our image. Let’s do that now.

$ git push

Navigate over to Hub in your browser and scroll down. You should see that a build was just triggered.

After the build finishes, navigate to the “Tags” tab and see that the image was updated.

Setup AutoTests

Excellent! We now have both our images building when we push to our source control repo. But this is just step one in our CI process. We should only push new images to the repository if all tests pass.

Docker Hub will automatically run tests if you have a docker-compose.test.yml file that defines a sut service. Let’s create this now and run our tests.

Open the projectz-svc project in your editor and create a new file name: docker-compose.test.yml and add the following yaml.

version: “3.6”

services:
sut:
build:
context: .
args:
NODE_ENV: test
ports:
– “8080:80″
command: npm run test

Commit the changes and push to GitHub.

$ git add docker-compose.test.yml
$ git commit -m “add docker-compose.test.yml for hub autotests”
$ git push origin master

Now navigate back to Hub and the projectz-svc repo. Once the build finishes, click on the build link and scroll to the bottom of the build logs. There you can see that the tests were run and the image was pushed to the repo.

If the build fails, you will see that the status turns to FAILURE and you will be able to see the error in the build logs.

Conclusion

In part 2 of this series, we showed you how Docker Hub is one of the easiest ways to automatically build your images and run tests without having to use a separate CI system. If you’d like to go further you can take a look at: 

Speed Up Your Development Flow With These Dockerfile Best PracticesDocker Hub Essentials DockerTalkAutomate Developer Workflows and Increase Productivity with Docker Hub
The post How to Build and Test Your Docker Images in the Cloud with Docker Hub appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Multi-arch build and images, the simple way

“Build once, deploy anywhere” is really nice on the paper but if you want to use ARM targets to reduce your bill, such as Raspberry Pis and AWS A1 instances, or even keep using your old i386 servers, deploying everywhere can become a tricky problem as you need to build your software for these platforms. To fix this problem, Docker introduced the principle of multi-arch builds and we’ll see how to use this  and put it into production.

Quick setup

To be able to use the docker manifest command, you’ll have to enable the experimental features.

On macOS and Windows, it’s really simple. Open the Preferences > Command Line panel and just enable the experimental features.

On Linux, you’ll have to edit ~/.docker/config.json and restart the engine.

Under the hood

OK, now we understand why multi-arch images are interesting, but how do we produce them? How do they  work?

Each Docker image is represented by a manifest. A manifest is a JSON file containing all the information about a Docker image. This includes references to each of its layers, their corresponding sizes, the hash of the image, its size and also the platform it’s supposed to work on. This manifest can then be referenced by a tag so that it’s easy to find.

For example, if you run the following command, you’ll get the manifest of a non-multi-arch image in the rustlang/rust repository with the nightly-slim tag:

$ docker manifest inspect –verbose rustlang/rust:nightly-slim{   “Ref”: “docker.io/amd64/rust:1.42-slim-buster”,  “Descriptor”: {    “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,    “digest”: “sha256:1bf29985958d1436197c3b507e697fbf1ae99489ea69e59972a30654cdce70cb”,    “size”: 742,    “platform”: {      “architecture”: “amd64”,      “os”: “linux”    }  },  “SchemaV2Manifest”: {    “schemaVersion”: 2,    “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,    “config”: {      “mediaType”: “application/vnd.docker.container.image.v1+json”,      “size”: 4830,      “digest”: “sha256:dbeae51214f7ff96fb23481776002739cf29b47bce62ca8ebc5191d9ddcd85ae”    },    “layers”: [      {      “mediaType”: “application/vnd.docker.image.rootfs.diff.tar.gzip”,      “size”: 27091862,      “digest”: “sha256:c499e6d256d6d4a546f1c141e04b5b4951983ba7581e39deaf5cc595289ee70f”
      },      {
        “mediaType”: “application/vnd.docker.image.rootfs.diff.tar.gzip”,        “size”: 175987238,        “digest”: “sha256:e2f298701fbeb02568c3dcb9822f8488e24ef12f5430bc2e8562016ba8670f0d”
      }    ]  }}

The question is now, how can we put multiple Docker images, each supporting a different architecture, behind the sametag?

What if this manifest file contained a list of manifests, so that the Docker Engine could pick the one that it matches at runtime? That’s exactly how the manifest is built for a multi-arch image. This type of manifest is called a manifest list.

Let’s take a look at a multi-arch image:

$ docker manifest inspect –verbose rust:1.42-slim-buster[  {    “Ref”: “docker.io/library/rust:1.42-slim-buster@sha256:1bf29985958d1436197c3b507e697fbf1ae99489ea69e59972a30654cdce70cb”,    “Descriptor”: {      “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,       “digest”: “sha256:1bf29985958d1436197c3b507e697fbf1ae99489ea69e59972a30654cdce70cb”,      “size”: 742,      “platform”: {         “architecture”: “amd64”,        “os”: “linux”      }    },    “SchemaV2Manifest”: { … }  },  {    “Ref”: “docker.io/library/rust:1.42-slim-buster@sha256:116d243c6346c44f3d458e650e8cc4e0b66ae0bcd37897e77f06054a5691c570”,    “Descriptor”: {      “mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,      “digest”: “sha256:116d243c6346c44f3d458e650e8cc4e0b66ae0bcd37897e77f06054a5691c570”,      “size”: 742,      “platform”: {        “architecture”: “arm”,        “os”: “linux”,        “variant”: “v7”      }    },    “SchemaV2Manifest”: { … }…]

We can see that it’s a simple list of the manifests of all the different images, each with a platform section that can be used by the Docker Engine to match itself to.

How they’re made

There are two ways to use Docker to build a multiarch image: using docker manifest or using docker buildx.

To demonstrate this, we will need a project to play. We’ll use the following Dockerfile which just results in a Debian based image that includes the curl binary.

ARG ARCH=FROM ${ARCH}debian:buster-slimRUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*ENTRYPOINT [ “curl” ]

Now we are ready to start building our multi-arch image.

The hard way with docker manifest

We’ll start by doing it the hard way with `docker manifest` because it’s the oldest tool made by Docker to build multiarch images.

To begin our journey, we’ll first need to build and push the images for each architecture to the Docker Hub. We will then combine all these images in a manifest list referenced by a tag.

# AMD64$ docker build -t your-username/multiarch-example:manifest-amd64 –build-arg ARCH=amd64/ .$ docker push your-username/multiarch-example:manifest-amd64# ARM32V7$ docker build -t your-username/multiarch-example:manifest-arm32v7 –build-arg ARCH=arm32v7/ .$ docker push your-username/multiarch-example:manifest-arm32v7# ARM64V8$ docker build -t your-username/multiarch-example:manifest-arm64v8 –build-arg ARCH=arm64v8/ .$ docker push your-username/multiarch-example:manifest-arm64v8

Now that we have built our images and pushed them, we are able to reference them all in a manifest list using the docker manifest command.

$ docker manifest create your-username/multiarch-example:manifest-latest –amend your-username/multiarch-example:manifest-amd64 –amend your-username/multiarch-example:manifest-arm32v7 –amend your-username/multiarch-example:manifest-arm64v8

Once the manifest list has been created, we can push it to Docker Hub.

$ docker manifest push your-username/multiarch-example:manifest-latest

If you now go to Docker Hub, you’ll be able to see the new tag referencing the images:

The simple way with docker buildx

You should be aware that buildx is still experimental.

If you are on Mac or Windows, you have nothing to worry about, buildx is shipped with Docker Desktop. If you are on linux, you might need to install it by following the documentation here https://github.com/docker/buildx

The magic of buildx is that the whole above process can be done with a single command.

$ docker buildx build –push –platform linux/arm/v7,linux/arm64/v8,linux/amd64 –tag your-username/multiarch-example:buildx-latest .

And that’s it, one command, one tag and multiple images.

Let’s go to production

We’ll now try to target the CI and use GitHub Actions to build a multiarch image and push it to the Hub.

To do so, we’ll write a configuration file that we’ll put in .github/workflows/image.yml of our git repository.

name: build our imageon:  push:    branches: masterjobs:  build:    runs-on: ubuntu-latest    steps:      – name: checkout code        uses: actions/checkout@v2      – name: install buildx        id: buildx        uses: crazy-max/ghaction-docker-buildx@v1        with:          version: latest      – name: build the image      run: |        docker buildx build           –tag your-username/multiarch-example:latest           –platform linux/amd64,linux/arm/v7,linux/arm64 .

Thanks to the GitHub Action crazy-max/docker-buildx we can install and configure buildx with only one step.

To be able to push, we now have to get an access token on Docker Hub in the security settings.

Once you created it, you’ll have to set it in your repository settings in the Secrets section. We’ll create DOCKER_USERNAME and DOCKER_PASSWORD variables to login afterward.

Now, we can update the GitHub Action configuration file and add the login step before the build. And then, we can add the –push to the buildx command.

…      – name: login to docker hub        run: echo “${{ secrets.DOCKER_PASSWORD }}” | docker login -u “${{ secrets.DOCKER_USERNAME }}” –password-stdin      – name: build the image        run: |          docker buildx build –push             –tag your-username/multiarch-example:latest             –platform linux/amd64,linux/arm/v7,linux/arm64 .

We now have our image being built and pushed each time something is pushed on master.

Conclusion

This post gives an example of how to build a multiarch Docker image and push it to the Docker Hub. It also showed how to automate this process for git repositories using GitHub Actions; but this can be done from any other CI system too.

An example of building multiarch image on Circle CI, Gitlab CI and Travis can be found here.
The post Multi-arch build and images, the simple way appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/