Compiling Qt with Docker Using Caching

This is a guest post from Viktor Petersson, CEO of Screenly is the most popular digital signage product for the Raspberry Pi. Find Viktor on Twitter @vpetersson.

In the previous blog post, we talked about how we compile Qt for Screenly OSE using Docker’s nifty multi-stage and multi-platform features. In this article, we build on this topic further and zoom in on caching. 

Docker does a great job with caching using layers. Each command (e.g., RUN, ADD, etc.) generates a layer, which Docker then reuses in future builds unless something changes. As always, there are exceptions to this process, but this is generally speaking true. Another type of caching is caching for a particular operation, such as compiling source code, inside a container.

At Screenly, we created a Qt build environment inside a Docker container. We created this Qt build to ensure that the build process was reproducible and easy to share among developers. Since the Qt compilation process takes a long time, we leveraged ccache to speed up our Qt compilation. Implementing ccache requires volume mounting a folder from outside of the Docker environment. 

The above steps work well if you are the only developer working on the project. What happens if you want to be able to have a shared cache across a team?

There are a few ways to accomplish this style of caching in Docker. 

The easiest way to establish a shared cache is by following what we did in the previous article. We used disk cache along with some neat features for speeding up caching in BuildKit. We then compressed the cached files and distributed them to team members. The process is not very elegant, but it gets the job done. 

If we want to automate the process a bit further, we can include retrieving the cache as part of the build process. An example of this could looks like this:

RUN curl -o /tmp/build-cache.tgz &&
tar xfz /tmp/build-cache.tgz -C /tmp &&
rm /tmp/build-cache.tgz

This process above is neat, but it does mean that someone will need to periodically upload the build cache in order to keep cache files fresh. In addition, you need somewhere to store the files (such as S3).

It would be nice if we could avoid manual tasks and use native Docker technologies to do the same thing, right? As it turns out, we can use Docker to improve the process. We just need to use our imagination. 

As we showed in the previous article, we can use multi-stage builds to copy data between different docker images. What if we move the cache to a dedicated Docker image? We can then push this image to Docker Hub and pull it into the build process. 

The process is straightforward. Start by creating two different images in Docker Hub. Call them screenly/build-cache and screenly/build-env. Building on the previous article, we use this Dockerfile as the basis for screenly/build-env.

In the Dockerfile, we set the environment variable CCACHE_DIR to /src/ccache. This step tells ccache that the cache resides in /src/ccache. In the previous post, the step was just a volume mount to the system. However, in this case, we want to alter this step so that the cache lives outside of /src, as this is used for volume mounting the code base, such as /usr/ccache.

We can now launch the container with:

$ docker run –rm -t
-v ~/tmp/qt-src:/src
-v ~/tmp/qt-build:/build
-v ~/tmp/ccache:/usr/ccache

After you have run through your compilation, you can now build and push our cache image. The final Dockerfile will look like this:

FROM scratch
COPY ccache /ccache

To build this image, use the following code:

$ cd ~/tmp
$ docker build
-f /path/to/Dockerfile
-t screenly/build-cache
$ docker push screenly/build-cache

Finally, you can now include this layer in the Dockerfile for screenly/build-env. Add the line:

COPY –from=screenly/build-cache /ccache /usr/ccache

Next time you rebuild screenly/build-env, Docker will automatically pull down the cache. Also, you only need to add volume mounting when you are refreshing the cache. 

About Screenly

Screenly is the most popular digital signage product for the Raspberry Pi. If you want to turn a physical screen into a secure, remote-managed device (over UI or digital signage API), Screenly makes setup a breeze. Users can display beautiful dashboards, images, videos, and web page content.

Screenly is available in two flavors: an open source version and a commercial version. You can try out our commercial version with a 14-day free trial.
The post Compiling Qt with Docker Using Caching appeared first on Docker Blog.
Quelle: – Service Mesh Feature Comparison — including Istio, Linkerd 2, AWS App Mesh, Consul, Maesh, Kuma, Open Service Mesh (OSM)

New Docker Desktop Preview for Apple M1 Released

This is just a quick update to let you know that we’ve released another preview of Docker Desktop for Apple M1 chips, which you can download from our Docker Apple M1 Tech Preview page. The most exciting change in this version is that Kubernetes now works.

First, a big thank you to everyone who tried out the previous preview and gave us feedback. We’re really excited to see how much enthusiasm there is for this, and also really grateful to you for reporting what doesn’t yet work and what your highest priorities are for quick fixes. In this post, we want to update you on what we’ve done and what we’re still working on.

Some of the biggest things we’ve been doing since the New Year are not immediately visible but are an essential part of eventually turning this into a supported product. The previous preview was built on a developer’s laptop from a private branch. Now all of the code is fully integrated into our main development branch. We’ve extended our CI suite to add several M1 machines, and we’ve extended our CI code to build and test Docker Desktop itself and all our dependencies for both architectures in parallel. With the whole pipeline now automated, from now on we will be able to issue new previews on a more regular basis and have more confidence that our changes have not broken anything.

As for feature changes and bug fixes since the last preview, here are some of the highlights:

Kubernetes now works (although you might need to reset the cluster in our Troubleshoot menu one time to regenerate the certificates).The host.docker.internal and vm.docker.internal DNS entries now resolve.We removed hard-coded IP addresses: it now dynamically discovers the IP allocated by macOS.osxfs file sharing now works.We made a configuration change that should improve disk performance.The Restart option in the Docker menu works.

The last major thing that we’re still working on is:

HTTP proxy support. At the moment the HTTP proxy configured on the host is ignored.

Finally, we are aware of the following items which are unfortunately out of our control. Here are our best recommendations for now:

Some corporate security or VPN software blocks the connection between the host and the VM, or the VM and the outside world. This can happen even if it doesn’t happen on Intel Macs because we had to switch to a new connection method with Apple’s new virtualization framework. There are some possible workarounds posted by users on our github issue, you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.

If you have an M1 Mac, then we invite you to download this new build and try it out. (Just bear in mind that it’s still a preview, so expect some rough edges.) If you encounter any bugs, please let us know on our GitHub repo. If you filed a bug against the previous preview, now would be a good time to retest it and let us know either that it’s now fixed or that it isn’t. You can chat with other users on the #docker-desktop-mac channel on our community Slack. And finally, if you’re the sort of user who wants to be the first to try out early versions of our software (not just M1) we invite you to join our Developer Preview Program.
The post New Docker Desktop Preview for Apple M1 Released appeared first on Docker Blog.

Analyze AWS EKS Audit logs with Falco – Falco is a Kubernetes threat detection engine. Falco supports Kubernetes Audit Events to track the changes defined in k8s audit rules made to your cluster. But unfortunately, AWS EKS is a managed…

Helm — Kubernetes Package Manager – In Docker, we build docker images and store them in a remote repository such as Docker Hub, GCR or ECR. Same with Helm you can generate Kubernetes packages with Helm called Charts. These charts can…

Kata Container And GVisor With K0s – As not all pods can be trusted, this article will show different options to enhance process isolation through the usage of container runtimes other than the default one (runc). We will use Kubernetes…