Simple logging with ECK and Fluentd
Centrailize your log storageContinue reading on Kubernauts »
Quelle: blog.kubernauts.io
Hey Docker run EKS, with Auto Fleet Spotting Support, please!
Photo by Hamish Weir on UnsplashUpdates:2020/08/27: Git repo updated with only private subnets support option.IntroductionWe define Auto Fleet Spotting as a way to provide support for Auto Scaling of a Fleet of Spot Instances on AWS EKS.This implementation is based on the official upstream Terraform AWS EKS implementation and was extended to provide an easy way for the deployment of EKS clusters with Kubernetes 1.17.9 in any region with Auto Fleet Spotting support.With this simple AWS EKS deployment automation solution with Docker you’ll not only be able to save time by EKS deployments and enforce some security best practices, but also you can reduce your Kubernetes costs by up to 90%.I’m not joking, we’re using this implementation for developing and running our own Kubernautic Platform, which is the basis of our Rancher Shared as a Service offering.We’re using the same implementation for other dedicated Kubernetes clusters running on AWS EKS which are running with a ratio of 98% for our Spot to On-Demand Instances and scale roughly from 6 to 40 Spot Instances during the day. Here is the Grafana view on the node count of a real world project over the last 7 days.TL;DRIf you’d like to try this implementation, all what you need is to clone the extended AWS Terraform EKS repo and run the docker/eks container to deploy the latest EKS Kubernetes 1.17.9 version (at this time of writing at 20/08/16). If you’d like to build your own image, please refer to this section of the README file in the repo.# Do this on your own risk, if you might want to trust me :-)$ git clone https://github.com/arashkaffamanesh/terraform-aws-eks.git& cd terraform-aws-eks$ docker run -it –rm -v "$HOME/.aws/:/root/.aws" -v "$PWD:/tmp" kubernautslabs/eks -c "cd /tmp; ./2-deploy-eks-cluster.sh"Run EKS without the Docker ImageIf these Prerequisites are met on your machine, you can run the deployment script directly without running the docker container:$ ./2-deploy-eks-cluster.shWould you like to have a cup of coffee, tee or a cold German beer?The installation should take about 15 minutes, time enough to enjoy a delicious cup of coffee, tee or a cold German beer ;-)Please let me describe what the deployment scripts mainly do and how you can extend the cluster with a new worker node group for On-Demand or additional Spot Instances and deploy additional add-ons like Prometheus, HashiCorp Vault, Tekton for CI/CD and GitOps, HelmOps, etc.The first script `./1-clone-base-module.sh` copies the base folder from the root of the repo to a new folder which is the cluster name which you provide after running the script and provides a README file in the cluster folder with the needed steps to run the EKS deployment step by step.The second script `2-deploy-eks-cluster.sh` makes the life easier and automates all the steps which is needed to deploy an EKS cluster. The main work which the deployment script does is to ask you about the cluster name and region and apply some few sed operations to substitute some values in the configuration files like setting the S3 bucket name, the cluster name and region and create a key pair, the S3 bucket and then run `make apply` to deploy the cluster with terraform and deploy the cluster autoscaler by setting the autoscaling group name in the cluster-autoscaler-asg-name.yml.Why using S3?The S3 bucket is very useful, if you want to work with your team to extend the cluster, it mainly saves the cluster state file `terraform.tfstate` in the S3 bucket, the S3 bucket should have a unique name and will be defined as the cluster name in the backend.tf file:https://medium.com/media/cc96ff9a3a086aaf41b1262a673d7070/hrefOne of the essential parts to get autoscaling working with spot instances is to define the right IAM Policy Document and attach it to the worker autoscaling group of the worker node group IAM role:https://medium.com/media/4a26caa805dd79cdb52ed85ad56fa92a/hrefHow does the whole thing work?Each cluster may have one or more Spot Instances or On-Demand Worker groups, sometimes referred as node pools as well. The initial installation will deploy only one worker group named spot-1, which we can easily extend after the initial deployment and run make apply again.If you’d like to add a new On-Demand worker group, you’ve to extend the `main.tf` file in your cluster module (in terraform a folder is named module) as shown below and run make plan followed with make apply:$ make plan$ make applyhttps://medium.com/media/6a117e59f75001d9794bd3edfed50df8/hrefUsing Private SubnetsI extended the VPC module definition to make use of private subnets with a larger /20 network address range, which allows us to use up-to 4094 IPs for our pods.https://medium.com/media/98f725c41df79cb9caeace191ca538eb/hrefAdd-OnsSome additional components which we refer as add-ons are provided in the addons folder in the repo. Each addon should have a README file about how to use it.TestingTo test how the auto-scaling of spot instances works, you can deploy the provided nginx-to-scaleout.yaml from the addons folder and scale it to 10 replicas and see how the spot instances are scaled-out and decrease the number of replicas to 1 and see how the spot instances are scaled-in back (the scaled-in process may take up to 5 minutes or longer, or even don't happen, if something was not configured properly!).k create -f ../addons/nginx-to-scaleout.yamlk scale deployment nginx-to-scaleout –replicas=10k get pods -wk get nodesConclusionRunning an EKS cluster with autoscaling support for spot instances or on-demand instances is a matter of running a single docker run command, even if you don’t have terraform, the aws-iam-authenticator or the kubectl-terraform plugin installed. All needed configurations are provided as code, we didn’t even had to login to the AWS web console to setup the EKS cluster and configure the permissions for the cluster auto-scaler for for our worker groups.This is the true Infrastructure as Code made with few lines of code and minor adaptions to the upstream Terraform EKS implementation to run EKS with support of Autoscaling of Spot and On-Demand instances.We are using this implementation with Kubernetes 1.15 today, the in-place migration from 1.15 directly to 1.17 can’t work directly, we’re going to use blue green deployments with traffic routing to migrate our workloads and our users who are using our Kubernautic Platform to a new environment, users would be able to use a full-fledged, fully isolated virtual Kubernetes cluster running in a Stateful Namespace, stay tuned!Questions?Join us on Kubernauts Slack channel or add a comment here.We’re hiring!We are looking for engineers who love to work in Open Source communities like Kubernetes, Rancher, Docker, etc.If you wish to work on such projects and get involved to build the NextGen Kubernautic Platform, please do visit our job offerings page or ping us on twitter.Related resourcesTerraform AWS EKSTerraform Provider EKSHey Docker run EKS, with Auto Fleet Spotting Support, please! was originally published in Kubernauts on Medium, where people are continuing the conversation by highlighting and responding to this story.
Quelle: blog.kubernauts.io
Deploy Ghost in a Spot Namespace on RSaaS from your browser
Deploy Ghost in a Spot Namespace on Kubernautic from your browserPhoto by Karim MANJRA on UnsplashIn my previous post Spot Namespaces was introduced.In this post I’m going to show you how you can run ghost, a headless CMS or your own apps in a namespace by defining the proper requests and limits in the deployment manifest file of ghost or your own app on Kubernautic.After you’re logged into Rancher you’ll have access to your project on your assigned Kubernetes cluster in form of <user-id>-project-spot-namespace:Click on the project and in the navigation menu bar click on Namespaces, now you can see a the namespace which is assigned to your project:Now that we know the id of our namespace, we’re ready to deploy our Ghost app.First you need to select the cluster and click on the “Launch kubectl” button:With that we’ll have Shell access to the cluster in our browser:With the following kubectl command, you can switch to your namespace:kubectl config set-context –current –namespace <your namespace>And deploy Ghost like this:kubectl create -f https://raw.githubusercontent.com/kubernauts/practical-kubernetes-problems/master/3-ghost-deployment.yamlPlease note in the spec part of the manifest we’re setting the limits for CPU and Memory in the resources part of our ghost image, unless you’ll not be able to deploy the app.spec: containers: – image: ghost:latest imagePullPolicy: IfNotPresent name: ghost resources: limits: memory: 512Mi cpu: 500m requests: memory: 256MiTo see if our deployment was successful, we can run:kubectl get deploymentNow we can create a L7 Ingress to ghost easily in Rancher by selecting Resources → Workloads → LoadBalancing → Add ingress.In this example we’re going to tell Rancher to generate a .xip.io hostname for us, we select the target which is ghost and provide the port 2368 in Port filed:After couple of seconds the L7 ghost ingress is ready to go, click on the ghost-ingress-xyz-xip.io link and enjoy:Questions?If you have any questions, we’d love to welcome you to our Kubernauts Slack channel and be of help.We’re hiring!We love people who love building things, New Things!If you wish to work on New Things, please do visit our job offerings page.Deploy Ghost in a Spot Namespace on RSaaS from your browser was originally published in Kubernauts on Medium, where people are continuing the conversation by highlighting and responding to this story.
Quelle: blog.kubernauts.io
Hello HelmOps with Tekton and Chartmuseum on Kubernetes
Photo by Jonathan Singer on UnsplashThis tutorial should help you to get started with Helm Operations, referred to as HelmOps and CI/CD with Tekton on K3s running on your Laptop deployed on multipass Ubuntu VMs with MetalLB support. We’ll use Chartmuseum to store a simple Helm Chart in our own Helm Repository in Chartmuseum and deploy our apps from there through Tekton Pipelines in the next section of this tutorial.This guide should work with any K8s implementation running locally or on any cloud with some minor adaptations to the scripts provided in the Github Repo of Bonsai.To learn more about K3s and our somehow opinionated Bonsai implementation on Multipass VMs with MetalLB support, please refer to this article about K3s with MetalLB on Multipass VMs.Get startedWith the following commands you should get a K3s cluster running on your machine in about 3 minutes on real Ubuntu VMs, which we refer sometimes as a near-to-production environment.git clone https://github.com/kubernauts/bonsai.gitcd bonsai./deploy-bonsai.sh# please accept all default values (1 master and 2 workers)Deploy ChartmuseumPrerequisitesA running k8s clusterLB service (it will be installed by default with Bonsai)helm3 versionWe deploy chartmuseum with custom values and create an ingress with the chartmuseum.local domain name.Please set the following host entries in your /etc/hosts file:# please adapt the IP below with the external IP of the traefik LB (run `kubectl get svc -A` and find the external IP)192.168.64.26 chart-example.local192.168.64.26 chartmuseum.local192.168.64.26 tekton-dashboard.local192.168.64.26 registry-ui.local192.168.64.26 getting-started-triggers.localAnd deploy chartmuseum:cd addons/chartmuseumhelm repo add stable https://kubernetes-charts.storage.googleapis.comkubectl create ns chartmuseumkubectl config set-context –current –namespace chartmuseumhelm install chartmuseum stable/chartmuseum –values custom-values.yamlkubectl apply -f ingress.yamlNow you should be able to access chartmuseum through:http://chartmuseum.localIf you see the “Welcome to ChartMuseum!”, then you’re fine.Now we are ready to add the chartmuseum repo and install the helm-push plugin and package the sample hello-helm chart and push it to our chartmuseum repo running on K3s:# add the repohelm repo add chartmuseum http://chartmuseum.local# install helm push plugin:helm plugin install https://github.com/chartmuseum/helm-push.git# build the package:cd hello-helm/charthelm package hello/# the helm package name will be hello-1.tgzls -l hello-1.tgz# push the package to chartmuseumhelm push hello-1.tgz chartmuseum# You should get:Pushing hello-1.tgz to chartmuseum…Done.Install the ChartTo install a chart, this is the basic command used:helm install <chartmuseum-repo-name>/<chart-name> –name <release-name> (helm2)helm install <release-name> <chartmuseum-repo-name>/<chart-name> (helm3)We need to update our helm repos first and install the chart:helm repo updatehelm install hello chartmuseum/helloWe should get a similar output like this:NAME: helloLAST DEPLOYED: Sat Jul 25 17:56:23 2020NAMESPACE: chartmuseumSTATUS: deployedREVISION: 1TEST SUITE: NoneHarvest your workcurl http://chart-example.local/Welcome to Hello Chartmuseum for Private Helm Repos on K3s Bonsai!What happened here?We deployed the chart with helm through the command line, the hello service was defined in the values.yaml with the type LoadBalancer and received an external IP from MetalLB.https://medium.com/media/262c126dd063c5258cd7e13a936c6f93/hrefIn the above values.yaml we define to have an ingress and the hostname which is chart-example.local as the domain name which is mapped in our /etc/hosts file to the IP address of the traefik load balancer, alternatively we could map the domain to the external IP address of the hello-hello service (in my case 192.168.64.27) as well.Delete the chart and the helm hello releaseSince we want to automate the deployment process in the next section with Tekton Pipelines and do the CI part as well, we’ll clean up the deployment and delete our chart from chartmuseum.helm delete hellocurl -i -X DELETE http://chartmuseum.local/api/charts/hello/1CI/CD and HelmOps on KubernetesIn this section, we’re going to introduce Tekton and Tekton Pipelines which is the technological successor to Knative Build and provides a Kube-Native style for declaring CI/CD pipelines on Kubernetes. Tekton supports many advanced CI/CD patterns, including rolling, blue/green, and canary deployment. To learn more about Tekton, please refer to the official documentation page of Tekton and also visit the new neutral home for the next generation of continuous delivery collaboration by Continuous Delivery Foundation (CDF).For the impatientKatakoda provides a very nice Scenario for Tekton which demonstrates building, deploying, and running a Node.js application with Tekton on Kubernetes using a private docker registry.By going through the Katakoda scenario in about 20 minutes, you should be able to learn about Tekton concepts and how to define various Tekton resources like Tasks, Resources and Pipelines to kick off a process through PipelineRuns to deploy our apps in a GitOps style manner on Kubernetes.In this section, we’re going to extend that scenario with Chartmuseum from the first section and provide some insights about Tekton Triggers which rounds up this tutorial and hopefully helps you to use it for your daily projects.About GitOps and xOpsGitOps is buzzing these days after the dust around DevOps, DevSecOps, NoOps, or xOps has settled down. We asked the community about the future of xOps and what’s coming next, if you’d like to provide your feedback, we’d love to have it:body[data-twttr-rendered=”true”] {background-color: transparent;}.twitter-tweet {margin: auto !important;} — @kubernautsfunction notifyResize(height) {height = height ? height : document.documentElement.offsetHeight; var resized = false; if (window.donkey && donkey.resize) {donkey.resize(height);resized = true;}if (parent && parent._resizeIframe) {var obj = {iframe: window.frameElement, height: height}; parent._resizeIframe(obj); resized = true;}if (window.location && window.location.hash === “#amp=1″ && window.parent && window.parent.postMessage) {window.parent.postMessage({sentinel: “amp”, type: “embed-size”, height: height}, “*”);}if (window.webkit && window.webkit.messageHandlers && window.webkit.messageHandlers.resize) {window.webkit.messageHandlers.resize.postMessage(height); resized = true;}return resized;}twttr.events.bind(‘rendered’, function (event) {notifyResize();}); twttr.events.bind(‘resize’, function (event) {notifyResize();});if (parent && parent._resizeIframe) {var maxWidth = parseInt(window.frameElement.getAttribute(“width”)); if ( 500 < maxWidth) {window.frameElement.setAttribute("width", "500");}}The reality is we did CVS- and SVN- Ops about 15 years ago with CruiseControl for both CI and CD. Later Hudson from the awesome folks at Sun Micorsystems was born and renamed to Jenkins, which is still one of the most widely used CI/CD tools today.Times have changed and with the rise of Cloud Computing, Kubernetes and the still buzzing term Cloud-Native, which is a new way of thinking, the whole community is bringing new innovations at the speed of light.Cloud Native: A New Way Of Thinking!New terms like Kube-Native-Ops, AppsOps, InfraOps and now HelmOps are not so common or buzzing yet at this time of writing. We believe that Tekton or Jenkins-X which is based on Tekton along with other nice solutions like ArgoCD, FluxCD, Spinnaker or Screwdriver are going to change the way we’re going to automate the delivery and roll-out of our apps and services the Kube-Native way on Kubernetes.With that said, let’s start with xOps and deploy our private docker registry and Tekton on K3s and run our simple node.js app from the previous section through helm triggered by Tekton Pipelines from Tekton Dashboard.Make sure you have helm3 version in place, add the stable and incubator Kubernetes Charts Helm Repo and install the docker registry proxy, the registry UI and the ingress for the registry UI:helm version –shorthelm repo add stable https://kubernetes-charts.storage.googleapis.comhelm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubatorhelm upgrade –install private-docker-registry stable/docker-registry –namespace kube-systemhelm upgrade –install registry-proxy incubator/kube-registry-proxy –set registry.host=private-docker-registry.kube-system –set registry.port=5000 –set hostPort=5000 –namespace kube-systemkubectl apply -f docker-registry-ui.yamlkubectl apply -f ingress-docker-registry-ui.yamlAccess the Docker Registryhttp://registry-ui.local/Deploy Tekton Resources, Triggers and Dashboardkubectl apply -f pipeline-release.yamlkubectl apply -f triggers-release.yamlkubectl config set-context –current –namespace tekton-pipelineskubectl apply -f tekton-dashboard-release.yamlkubectl apply -f ingress-tekton-dashboard.yamlAccess the Tekton DashboardThe Tekton dashboard for this installation on K3s can be reached viahttp://tekton-dashboard.local/The dashboard is great to see how tasks and PipelineRuns are running or if something goes wrong to get the logs for troubleshooting or sell it as your xOps Dashboard ;-)For now we don’t have any pipelines running. We’re going to run some tasks first, build a pipeline and initiate it in the next steps along with pipeline runs. But before we fire the next commands, I’d like to explain what we’re going to do.We have 2 apps, the first app is named app, which builds an image from the source on github with kaniko executor and pushes it to our private docker registry and deploy it from there into the hello namespace, where we deploy our hello app from our private chartmuseum repo for the HelmOps showcase as well.N.B.: we cleaned-up the same hello app at the end of the first section of this tutorial and are going to deploy it through HelmOps.Take it easy or not, we’re going to create a task, build a pipeline containing our task, use the TaskRun resource to instantiate and execute a Task outside of the Pipeline and use the PipelineRun to instantiate and run the Pipeline containing our Tasks, oh my God? ;-)Take it easy, let’s start with HelmOps or xOpsYou need the tkn cli to follow the next steps, I’m using the latest version at this time of writing.➜ pipeline git:(master) ✗ tkn versionClient version: 0.11.0Pipeline version: v0.15.0Triggers version: v0.7.0We create a namespace named hello and switch to the namespace:kubectl create ns hellokubectl config set-context –current –namespace helloNow we need to define the git-resource, which is the source in our git repo defined through a custom resource definition PipelineResource:https://medium.com/media/d3cfe785de60fce2658e90bfeb8bf36e/hrefAnd apply the git-resource:kubectl apply -f node-js-tekton/pipeline/git-resource.yamlWith that we are defining where our source is coming from in git, in our case from Github.We can now list our first resource with the tkn cli tool as follow:tkn resources listIn the next step, we’re going to define a task named “build-image-from-source”. In the spec part of the Task object (CRD) we define the git-source with the type git and some parameters as pathToContext, patToDockerfile, imageUrl and imageTag as inputs and the steps needed to run the task, the list-src (which is optional) and the build-and-push step which uses the kanico-project executor image with the command /kaniko/executor/ and some args to build and push the image.https://medium.com/media/d77e198facd1fbc89e8882dd49015ab9/hrefNow we need to deploy our task build and push Task resource to our cluster with:kubectl apply -f node-js-tekton/pipeline/task-build-src.yamlAnd deploy another Task which is defined in the task-deploy.yamlhttps://medium.com/media/c2fcf43d3fd076b7c5df2391cc2c5140/hrefThis Task has 4 steps:update-yaml # set the image tagdeploy-app # deploy the first apppush-to-chartmuseum # push the hello helm chart to chartmuseumhelm-install-hello # install the helm chart to the clusterThe last 3 tasks are using a slightly extended helm-kubectl image which has the helm push plugin installed.N.B. the Dockerfile for helm-kubectl is provided under addons/helm-kubectl.We can now run the task-deploy.yaml and list our tasks with:kubectl apply -f node-js-tekton/pipeline/task-deploy.yamltkn tasks listAnd finally we’ll declare the pipeline and run it with the pipeline-run entity.Nice to know: In PipelineRun we define a collection of resources that define the elements of our Pipeline which include tasks.kubectl apply -f node-js-tekton/pipeline/pipeline.yamltkn pipelines listkubectl apply -f node-js-tekton/pipeline/service-account.yamlkubectl get ServiceAccountskubectl apply -f node-js-tekton/pipeline/pipeline-run.yamltkn pipelineruns listtkn pipelineruns describe application-pipeline-runkubectl get all -n hellocurl chart-example.localWelcome to Hello Chartmuseum for Private Helm Repos on K3s Bonsai!Harvest your Helm-X-Ops workIf all steps worked, in the Tekton Dashboard you should get what you did in the hello namespace, enjoy :-)What’s coming next: AutomationIn the third section of this tutorial, we’ll see how to use Tekton Triggers to automate our HelmOps process, stay tuned.Related resourcesSample nodes-js app for tektonhttps://github.com/javajon/node-js-tektonTekton Pipelines on Githubhttps://github.com/tektoncd/pipelineKatacoda Scenarionshttps://www.katacoda.com/javajon/courses/kubernetes-pipelines/tektonhttps://katacoda.com/ncskier/scenarios/tekton-dashboardTekton Pipelines Tutorialhttps://github.com/tektoncd/pipeline/blob/master/docs/tutorial.mdOpen-Toolchain Tekton Catalogopen-toolchain/tekton-catalogGitOpshttps://www.gitops.tech/Building a ChatOps Bot With TektonBuilding a ChatOps Bot With TektonHello HelmOps with Tekton and Chartmuseum on Kubernetes was originally published in Kubernauts on Medium, where people are continuing the conversation by highlighting and responding to this story.
Quelle: blog.kubernauts.io
Kubernautic Data Platform for DataOps & MLOps
Kubernautic Data Platform base componentsIntroductionEvery digital transformation needs a data platform to transform data and master the challenges of Data Operations referred as DataOps to bring Data, Apps and Processes made by humans and machines together.To make data accessible in real time to the right data scientists by decoupling business decisions from the underlying infrastructure, organizations need to remove bottlenecks from their data projects by implementing a Data Platform based on the best DataOps practices.DataOps DefinedAccording to Wikipedia DataOps was first introduced by Lenny Liebmann, Contributing Editor, InformationWeek, in a blog post on the IBM Big Data & Analytics Hub titled “3 reasons why DataOps is essential for big data success” on June 19, 2014.DataOps is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics.DataOps Re-DefinedEckerson Group re-defines DataOps as follow, which we like so much:“DataOps is an engineering methodology and set of practices designed for rapid, reliable, and repeatable delivery of production-ready data and operations-ready analytics and data science models. DataOps enhances an advanced governance through engineering disciplines that support versioning of data, data transformations, data lineage, and analytic models. DataOps supports business operational agility with the ability to meet new and changing data and analysis needs quickly. It also supports portability and technical operations agility with the ability to rapidly redeploy data pipelines and analytic models across multiple platforms in on-premises, cloud, multi-cloud, and hybrid ecosystems.” [1]Take it easy: DataOps combines Agile development, DevOps and statistical process controls and applies them to data analytics, but needs a well designed Data Platform!What is a Data Platform?We define a Data Platform as the infrastructure with a minimal set, easy to extend base components to successfully deliver data-driven business outcomes with increased productivity and improved collaboration between data engineers, data and security operators through a governed Self-service operation with the highest possible automation.That said, we believe a governed Self-service operation and automation is key to the success of most DataOps initiatives and needs a Platform to build Platforms. With Kubernautic Platform we made it happen to build a Data Platform based on Kubernetes, which is a Platform to build Platforms.Kubernautic Data Platform: DataOps Orchestration on KubernetesKubernautic Data Platform provides a cloud-native infrastructure with the base components as the foundation to run DataOps initiatives on Kubernetes through orchestration in a Self-service manner without the need to operate the platform on top of our Kubernautic offering.Kubernautic Data Platform is designed to enable data scientists to deploy, implement and run their data analytic pipelines built on top of Apache Kafka, Cassandra, MQTT implementations, Spark and Flink with Jupyter notebooks to achieve the following goals:Gather and manage data in one secure placeBuild and share interactive dashboardsReduce time spent on errors and operational tasksAccelerate productivity through team collaborationOrchestrate your development and production pipelines for re-usability and securityUnlock the full potential of DataOps without the need to operate the platformCreate innovative data analytics and deliver intelligent business valueDataOps Needs a Culture made by People to build the Data FactoryLike DevOps, DataOps is mainly a principle to create and develop a strategy in your organisation to overcome the cultural obstacles to achieve higher agility by development and delivery of enterprise grade data pipelines to turn data into value through automation and the right processes and product design made by people to build the Data Factory!Kubernautic Data Platform: The DataOps Data FactoryThese days machines are assembled mostly by machines designed by people and complex processes in a data-driven world to deliver products in factories.The following quote from Elon Musk describes the true problem and the solution to thinking of the Factory like a product.“We realised that the true problem, the true difficulty, and where the greatest potential is — is building the machine that makes the machine. In other words, it’s building the factory. I’m really thinking of the factory like a product.” Elon MuskWith Kubernautic Data Platform we deliver a highly automated system to build a Data Factory for DataOps to build enterprise grade data-driven products and services on top of our Kubernautic Platform.Some base components of Kubernautic Data Platform — which mainly provides the persistent layer of the platform — are presented through the following figure:Automation and Self-service at the heart of Kubernautic Data PlatformLike our Kubernautic public offering, where developers can get instant access for free to Spot Namespaces on Kubernetes or B2B customers within minutes to a dedicated Kubernetes cluster managed by Rancher Cluster Manager, Kuberbautic Data Platform provides the same agile experience to access the Data Factory for data pipeline and model orchestration, test and deployment automation for data pipelines and analytic models to make decisions faster with high quality data governed across a range of users, use cases, architectures and deployment options.According to Gartner’s Survey Analysis in March 2020 titled as “Data Management Struggles to Balance Innovation and Control” [2], only 22% of a data team’s time is spent on new initiatives and innovation.The Self-service capability of our Data Platform allows DevOps teams to provision Kubernetes clusters with DataOps base components within a few hours, not days or weeks waiting for IT operation teams to set-up the environment to explore, blend, enrich and visualize data. And exactly this Self-service capability of Kubernautic Data Platform boosts innovation by your DataOps projects.DataOps and MLOps are almost the same, but …MLOps stands for Machine Learning Operations and is almost the sibling of DataOps.We love the great blog post by Fernando Velez — Chief Data Technologist by Persistent — in Data-Driven Business and Intelligence titled as:“DataOps and MLOps: Almost, But Not Quite The Same” [3]and invite you to read through Fernando’s view on this topic, which might help to keep your Data and Machine Learning initiatives accurate over time with NoOps!Related resources[1] DataOps: More Than DevOps for Data Pipelines (by Eckerson Group)https://www.eckerson.com/articles/dataops-more-than-devops-for-data-pipelines[2] Survey Analysis: Data Management Struggles to Balance Innovation and Controlhttps://www.gartner.com/en/documents/3982237/survey-analysis-data-management-struggles-to-balance-inn[3] DataOps and MLOps: Almost, But Not Quite The Same https://www.persistent.com/blogs/dataops-and-mlops-almost-but-not-quite-the-same/Kubernautic Data Platform for DataOps & MLOps was originally published in Kubernauts on Medium, where people are continuing the conversation by highlighting and responding to this story.
Quelle: blog.kubernauts.io
How Docker Desktop Networking Works Under the Hood
Modern applications make extensive use of networks. At build time it’s common to apt-get/dnf/yum/apk install a package from a Linux distribution’s package repository. At runtime an application may wish to connect() to an internal postgres or mysql database to persist some state, while also calling listen() and accept() to expose APIs and UIs over TCP and UDP ports. Meanwhile developers need to be able to work from anywhere, whether in an office or at home or on mobile or on a VPN. Docker Desktop is designed to ensure that networking “just works” for all of these use-cases in all of these scenarios. This post describes the tools and techniques we use to make this happen, starting with everyone’s favorite protocol suite: TCP/IP.
TCP/IP
When containers want to connect to the outside world, they will use TCP/IP. Since Linux containers require a Linux kernel, Docker Desktop includes a helper Linux VM. Traffic from containers therefore originates from the Linux VM rather than the host, which causes a serious problem.
Many IT departments create VPN policies which say something like, “only forward traffic which originates from the host over the VPN”. The intention is to prevent the host accidentally acting as a router, forwarding insecure traffic from the Internet onto secure corporate networks. Therefore if the VPN software sees traffic from the Linux VM, it will not be routed via the VPN, preventing containers from accessing resources such as internal registries.
Docker Desktop avoids this problem by forwarding all traffic at user-level via vpnkit, a TCP/IP stack written in OCaml on top of the network protocol libraries of the MirageOS Unikernel project. The following diagram shows the flow of packets from the helper VM, through vpnkit and to the Internet:
When the VM boots it requests an address using DHCP. The ethernet frame containing the request is transmitted from the VM to the host over shared memory, either through a virtio device on Mac or through a “hypervisor socket” (AF_VSOCK) on Windows. Vpnkit contains a virtual ethernet switch (mirage-vnetif) which forwards the request to the DHCP (mirage/charrua) server.
Once the VM receives the DHCP response containing the VM’s IP address and the IP of the gateway, it sends an ARP request to discover the ethernet address of the gateway (mirage/arp). Once it has received the ARP response it is ready to send a packet to the Internet.
When vpnkit sees an outgoing packet with a new destination IP address, it creates a virtual TCP/IP stack to represent the remote machine (mirage/mirage-tcpip). This stack acts as the peer of the one in Linux, accepting connections and exchanging packets. When a container calls connect() to establish a TCP connection, Linux sends a TCP packet with the SYNchronize flag set. Vpnkit observes the SYNchronize flag and calls connect() itself from the host. If the connect() succeeds, vpnkit replies to Linux with a TCP SYNchronize packet which completes the TCP handshake. In Linux the connect() succeeds and data is proxied in both directions (mirage/mirage-flow). If the connect() is rejected, vpnkit replies with a TCP RST (reset) packet which causes the connect() inside Linux to return an error. UDP and ICMP are handled similarly.
In addition to low-level TCP/IP, vpnkit has a number of built-in high-level network services, such as a DNS server (mirage/ocaml-dns) and HTTP proxy (mirage/cohttp). These services can be addressed directly via a virtual IP address / DNS name, or indirectly by matching on outgoing traffic and redirecting dynamically, depending on the configuration.
TCP/IP addresses are difficult to work with directly. The next section describes how Docker Desktop uses the Domain Name System (DNS) to give human-readable names to network services.
DNS
Inside Docker Desktop there are multiple DNS servers:
DNS requests from containers are first processed by a server inside dockerd, which recognises the names of other containers on the same internal network. This allows containers to easily talk to each other without knowing their internal IP addresses. For example in the diagram there are 3 containers: “nginx”, “golang” and “postgres”, taken from the docker/awesome-compose example. Each time the application is started, the internal IP addresses might be different, but containers can still easily connect to each other by human-readable name thanks to the internal DNS server inside dockerd.
All other name lookups are sent to CoreDNS (from the CNCF). Requests are then forwarded to one of two different DNS servers on the host, depending on the domain name. The domain docker.internal is special and includes the DNS name host.docker.internal which resolves to a valid IP address for the current host. Although we prefer if everything is fully containerized, sometimes it makes sense to run part of an application as a plain old host service. The special name host.docker.internal allows containers to contact these host services in a portable way, without worrying about hardcoding IP addresses.
The second DNS server on the host handles all other requests by resolving them via standard OS system libraries. This ensures that, if a name resolves correctly in the developer’s web-browser, it will also resolve correctly in the developer’s containers. This is particularly important in sophisticated setups, such as pictured in the diagram where some requests are sent over a corporate VPN (e.g. internal.registry.mycompany) while other requests are sent to the regular Internet (e.g. docker.com).
Now that we’ve described DNS, let’s talk about HTTP.
HTTP(S) proxies
Some organizations block direct Internet access and require all traffic to be sent via HTTP proxies for filtering and logging. This affects pulling images during build as well as outgoing network traffic generated by containers.
The simplest method of using an HTTP proxy is to explicitly point the Docker engine at the proxy via environment variables. This has the disadvantage that if the proxy needs to be changed, the Docker engine process must be restarted to update the variables, causing a noticeable glitch. Docker Desktop avoids this by running a custom HTTP proxy inside vpnkit which forwards to the upstream proxy. When the upstream proxy changes, the internal proxy dynamically reconfigures which avoids having to restart the Docker engine.
On Mac Docker Desktop monitors the proxy settings stored in system preferences. When the computer switches network (e.g. between WiFi networks or onto cellular), Docker Desktop automatically updates the internal HTTP proxy so everything continues to work without the developer having to take any action.
This just about covers containers talking to each other and to the Internet. How do developers talk to the containers?
Port forwarding
When developing applications, it’s useful to be able to expose UIs and APIs on host ports, accessible by debug tools such as web-browsers. Since Docker Desktop runs Linux containers inside a Linux VM, there is a disconnect: the ports are open in the VM but the tools are running on the host. We need something to forward connections from the host into the VM.
Consider debugging a web-application: the developer types docker run -p 80:80 to request that the container’s port 80 is exposed on the host’s port 80 to make it accessible via http://localhost. The Docker API call is written to /var/run/docker.sock on the host as normal. When Docker Desktop is running Linux containers, the Docker engine (dockerd in the diagram above) is a Linux program running inside the helper Linux VM, not natively on the host. Therefore Docker Desktop includes a Docker API proxy which forwards requests from the host to the VM. For security and reliability, the requests are not forwarded directly over TCP over the network. Instead Docker Desktop forwards Unix domain socket connections over a secure low-level transport such as shared-memory hypervisor sockets via processes labeled vpnkit-bridge in the diagram above.
The Docker API proxy can do more than simply forward requests back and forth. It can also decode and transform requests and responses, to improve the developer’s experience. When a developer exposes a port with docker run -p 80:80, the Docker API proxy decodes the request and uses an internal API to request a port forward via the com.docker.backend process. If something on the host is already listening on that port, a human-readable error message is returned to the developer. If the port is free, the com.docker.backend process starts accepting connections and forwarding them to the container via the process vpnkit-forwarder, running on top of vpnkit-bridge.
Docker Desktop does not run with “root” or “Administrator” on the host. A developer can use docker run –privileged to become root inside the helper VM but the hypervisor ensures the host remains completely protected at all times. This is great for security but it causes a usability problem on macOS: how can a developer expose port 80 (docker run -p 80:80) when this is considered a “privileged port” on Unix i.e. a port number < 1024? The solution is that Docker Desktop includes a tiny helper privileged service which does run as root from launchd and which exposes a “please bind this port” API. This raises the question: “is it safe to allow a non-root user to bind privileged ports?”
Originally the notion of a privileged port comes from a time when ports were used to authenticate services: it was safe to assume you were talking to the host’s HTTP daemon because it had bound to port 80, which requires root, so the admin must have arranged it. The modern way to authenticate a service is via TLS certificates and ssh fingerprints, so as long as system services have bound their ports before Docker Desktop has started – macOS arranges this by binding ports on boot with launchd – there can be no confusion or denial of service. Accordingly, modern macOS has made binding privileged ports on all IPs (0.0.0.0 or INADDR_ANY) an unprivileged operation. There is only one case where Docker Desktop still needs to use the privileged helper to bind ports: when a specific IP is requested (e.g. docker run -p 127.0.0.1:80:80), which still requires root on macOS.
Summary
Applications need reliable network connections for lots of everyday activities including: pulling Docker images, installing Linux packages, communicating with database backends, exposing APIs and UIs and much more. Docker Desktop runs in many different network environments: in the office, at home and while traveling on unreliable wifi. Some machines have restrictive firewall policies installed. Other machines have sophisticated VPN configurations. For all these use-cases in all these environments, Docker Desktop aims to “just work”, so the developer can focus on building and testing their application (rather than debugging ours!)
If building this kind of tooling sounds interesting, come and make Docker Desktop networking even better, we are hiring see https://www.docker.com/career-openings
DockerCon2022
Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post How Docker Desktop Networking Works Under the Hood appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
How Docker containers are supporting the COVID-19 genomic monitoring effort
This is a guest blog post from Dr. Kelsey Florek and Curtis Kapsak. A video presentation of this post from Docker’s Community All-Hands can be found here.
Dr. Kelsey Florek is the Senior Genomics and Data Scientist at the Wisconsin State Laboratory of Hygiene and a steering committee member of the State Public Health Bioinformatics group (StaPH-B).
Curtis Kapsak is a Bioinformatics Scientist at Theiagen Genomics and is a member of the State Public Health Bioinformatics group (StaPH-B).
The rapid appearance and global spread of a novel Severe Acute Respiratory Syndrome (SARS) virus in 2019 pushed public health laboratories to develop new methods for genomic monitoring efforts on a scale never seen before. Adding to this challenge, the approaches typically used in genomic data analysis often rely on cutting edge and often niche open source software and libraries that increase the complexity of setting up analytical pipelines or workflows. This along with a varying landscape of compute environments ranging from on-prem workstations to public Cloud created a significant barrier for many laboratories attempting to perform viral genomic monitoring.
Developing a library of dockerized tools
Public health laboratories inherently need to meet rigorous quality control and quality assurance standards. The tests performed in public health laboratories are either reported back to clinics to be used for patient care or used in aggregate to inform public health interventions or outbreak investigations. Analytical workflows are held to the same standards as other laboratory developed tests and to support this effort, the State Public Health Bioinformatics consortium (StaPH-B) started developing a repository of dockerized software that was commonly used in public health genomic data analyses, StaPH-B/docker-builds. This repository was designed to address the need for accessible software that is both highly reliable and reproducible. Combined with a usage guide, this repository provided a centralized location of maintained and tested open source tools to support laboratories developing analysis workflows.
Since its initial development in 2018 the StaPH-B/docker-builds repository has grown to contain multiple versions of over 90 different analytical tools from 19 different contributors, with several of the COVID-19 specific images achieving over 1 million pulls. Between March 2021 and January 2022 as more laboratories began genomic monitoring we saw a logarithmic increase in the number of Docker image pulls on core COVID-19 genomic analysis software.
Supporting COVID-19 genomic analytics
Bioinformatic pipelines or workflows consist of a variety of tools and often start from a form of raw or primary DNA sequencing data. These tools perform a variety of transformative or summary tasks and vary in both their computational requirements and dependencies. The process of sequencing the SARS-CoV-2 viral genome involves sectioning off the viral genome and sequencing small portions of the DNA in parallel. The result is a dataset containing hundreds of thousands to millions of short strings containing A’s, T’s, C’s, and G’s in a variety of sequence combinations. COVID-19 workflows then take these datasets, reconstruct the genome and use a variety of techniques to then characterize the virus.
Many laboratories across the globe have moved towards using a dedicated workflow language like WDL or Nextflow for their analytical workflows. Combining a workflow language with dockerized software allows for the creation and routine usage of workflows that are highly portable and easily adapted to a variety of compute environments. This gives laboratories the ability to run small datasets on a laptop or scale to a high performance compute cluster or cloud environment for large datasets. Additionally, these workflow approaches allow developing a modular analysis framework that enables swapping out software as new versions are released or issues are identified. With the rapid and constant evolution of the virus that causes COVID-19, updates to classification software are also frequently updating to maintain the ability to accurately identify variants.
Keeping up to speed with viral evolution
The COVID-19 virus evolves a bit slower than influenza accruing on average two mutations per month and different variants (Alpha, Delta, Omicron, etc.) are differentiated by various combinations of mutations. Classifying a virus requires constructing a phylogenetic tree that models the relationship of the new virus to other viruses. However, constructing a tree to compare each new virus to every previous virus is both computationally expensive and impractical. To address this, two commonly used methods have emerged including using a set of selected reference viruses to build a tree (Nextclade) or machine learning to classify mutational patterns (Pangolin). Both of these approaches require regular updates to ensure classification is occurring accurately with the most recent information. Leveraging containerization, StaPH-B has been able to maintain images with the latest models allowing users to run workflows knowing they are using the most up to date, robust, and tested classification tools.
Summary
The highly portable, scaleable, and efficient nature of containerization has transformed how public health disease monitoring is performed. The implementation of containerized workflows have enabled laboratories to quickly adopt complex analytical workflows, which in turn has grown the scale of the viral monitoring effort. The open source repository maintained by StaPH-B would not be possible without the community of bioinformaticians driving innovation. With more laboratories turning to sequencing and complex analytics there is a growing demand for people to bridge the gap between biology and informatics. If you are interested in a career in bioinformatics and using data to solve health problems be sure to check out the APHL-CDC Bioinformatics Fellowship!
DockerCon2022
Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post How Docker containers are supporting the COVID-19 genomic monitoring effort appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
File Sharing with Docker Desktop
One of the topics users of Docker Desktop often ask us about is file sharing. How do I see my source code inside my container? What’s the difference between a volume and a bind mount? Why is file sharing slower than on Linux, and how can I speed it up? In this blog post, I’ll cover the options you have, some tips and tricks, and finish with a sneak preview of what we’re currently working on.
Bind mounts and volumes
Suppose you run an Ubuntu container with docker run -it ubuntu bash. You’ll quickly find that (1) the container has its own filesystem, based on the filesystem in the Ubuntu image; (2) you can create, delete and modify files, but your changes are local to the container and are lost when the container is deleted; (3) the container doesn’t have access to any other files on your host computer.
So the natural next questions are, how can I see my other files? And how can my container write data that I can read later and maybe use in other containers? This is where bind mounts and volumes come in. Both of these use the -v flag to docker run to specify some files to share with the container.
The first option most people encounter is the bind mount, where part of your local filesystem is shared with the container. For example, if you run
docker run -it -v /users/stephen:/my_files ubuntu bash
then the files at /users/stephen will be available at /my_files in the container, and you can read and write to them there. This is very simple and convenient, but if you’re using Docker Desktop a named volume may have better performance, for reasons I’ll explain in the next section.
The second option, a named volume, is a filesystem managed by Docker. You can create a named volume with a command like docker volume create new_vol, and then share it into the container using the -v flag again:
docker run -it -v new_vol:/my_files ubuntu bash
These volumes persist even after the container has been deleted, and can be shared with other containers. You can also browse their contents from the Docker Desktop UI, using the Volumes tab that we added recently (and that is now free for all users including Docker Personal).
Performance considerations
To understand the performance differences between the options, we first have to talk briefly about how Docker Desktop works. Many people imagine that Docker Desktop is just a UI on top of some open source tools, but that’s only a small part of what it is. Docker Desktop is fundamentally an environment to develop and run Linux containers on your Mac or Windows machine, with seamless integration into the host so that it appears as if they’re running natively. It does this by setting up a Linux VM (or optionally a WSL 2 environment on Windows) in which to run the Docker engine and your containers, and then passing CLI commands, networking and files between the host and the VM.
Unfortunately it’s in the nature of virtualization that there is always an unavoidable small overhead in crossing the host-VM boundary. It’s only tiny, but in a development environment with a huge source tree and lots of reads and writes, it adds up, and can visibly affect performance. And because Docker Desktop does such a good job of hiding the underlying VM, it’s not obvious why this is happening. On Linux, the container has direct access to the bind-mounted filesystem, and because the implementation on Mac and Windows “feels native”, people intuitively expect the same performance there.
Named volumes do not suffer from the same problem because they are created inside the VM’s own filesystem, so they are as fast as on a Linux machine. In WSL 2, Windows manages file sharing rather than Docker managing it, but the same performance considerations apply: files mounted from the Windows file system can be slow, named volumes are fast, but in this case there is another option: files stored in the Linux filesystem are also inside the WSL VM so are fast too.
Best practices and tips
This gives us the main tip in optimizing performance. It’s convenient to use bind mounts at first, and you may find that they are fine for your use case. But if performance becomes a problem, then (1) make sure that you’re only sharing what you need to share, and (2) consider what could be shared in some other way than a bind mount. You have several options for keeping files inside the VM, including a named volume, Linux files in WSL, and the container’s own filesystem: which to use will depend on the use case. For example:
Source code that you are actively editing is an appropriate use of a bind mountLarge, static dependency trees or libraries could be moved into a named volume, or WSL, or even baked into the container imageDatabases are more appropriate in a named volume or WSLCache directories and logfiles should be put in a named volume or WSL (if you need to keep them after the container has stopped) or in the container’s own filesystem (if they can disappear when the container does).Files that the container doesn’t need shouldn’t be shared at all. In particular, don’t share the whole of your home directory. We have seen some people do this habitually so that they’ll always have access to whatever files they need, but unlike on Linux it’s not “free”.
One remaining option if you really need a bind mount for some large or high-traffic directory is a third-party caching/syncing solution, for example Mutagen or docker-sync. These essentially copy your files inside the VM for faster read/write access, and handle syncing (in one or both directions) between the copy and the host. But it involves an extra component to manage, so named volumes are still preferred where possible.
The future
We have used a variety of file sharing implementations over the years (Samba and gRPC FUSE on Windows Hyper-V; osxfs and gRPC FUSE on Mac; and Windows uses 9P on WSL 2). We have made some performance improvements over time, but none of them have been able to match native performance. But we are currently experimenting with a very promising new file sharing implementation based on virtiofs. Virtiofs is new technology that is specifically designed for sharing files between a host and a VM. It is able to make substantial performance gains by using the fact that the VM and the host are running on the same machine, not across a network. In our experiments we have seen some very promising results.
We have already released a preview of this technology for Docker Desktop for Mac, which you can get from our public roadmap (it requires macOS 12.2), and we are also planning to use it for the forthcoming Docker Desktop for Linux. We think that it will be able to make bind mounts a lot faster (although we would still recommend named volumes or the container’s own filesystem for appropriate use cases). We would love to hear your experiences of it.
Next steps
If you want to go into more depth about these topics, I recommend a talk that one of our Docker Captains, Jacob Howard, gave at DockerCon 2021 entitled A Pragmatic Tour of Docker Filesystems. It’s got loads of great information and practical advice packed into only 26 minutes!
To follow the progress of our current work on virtiofs, subscribe to the ticket on our public roadmap. That’s where we post preview builds, and we’d love you to try them and give us your feedback on them.
DockerCon2022
Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post File Sharing with Docker Desktop appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
Docker Captain Take 5 – Thomas Shaw
Docker Captains are select members of the community that are both experts in their field and are passionate about sharing their Docker knowledge with others. “Docker Captains Take 5” is a regular blog series where we get a closer look at our Captains and ask them the same broad set of questions ranging from what their best Docker tip is to whether they prefer cats or dogs (personally, we like whales and turtles over here). Today, we’re interviewing Thomas Shaw, one of our Docker Captains. He works as a Principal Automation Engineer at DIGIT Game Studios and is based in Ireland.
How/when did you first discover Docker?
I remember it like it was yesterday. he date was August 23rd 2013 and I was working as a Build Engineer for Demonware. During a visit to the Vancouver office, one of the developers named Mohan Raj Rajamanickham mentioned “a cool new tool” called Docker. It sounded too good to be true but I downloaded version 0.5 the next day while waiting for the flight back home to Dublin, Ireland. I played with the Docker CLI for several hours on the flight and that was it. Before the plane had touched the runway in Dublin I was completely sold on the potential of Docker. It solved an immediate problem faced by developers and build engineers alike, dependency hell.
Over the following 12 months we replaced our bare metal build agents with containerized build agents. It was a primitive approach at first. We built a CentOS 5 VM, tarred it up and created a container image from it. This was the base image on which we ran builds and tests over the next 2 years. We went from 8 bare metal build agents, each unique, each manually setup, each with different tooling versions to 4 build agents with just Docker installed.
It was a simple approach but it eliminated the management of several unique build agents. We saw a number of other benefits too, such as better build stability, increased build throughput by 300% and most importantly teams now owned their own dependencies. This approach worked well and around 2015 we started looking at moving our CI/CD pipelines into AWS. We originally took a hybrid approach and ran the majority of builds and tests in our own datacenter and just a handful in AWS. This was easier than expected. Docker made our workloads portable and we began to leverage the scalability of AWS for running tests. The unit tests (which were actually functional tests) for one of our main projects was taking over 1 hour per commit. Using containers we were able to split the tests across multiple containers on multiple build agents and we reduced the execution time to around 10 minutes. It was at this point that more folks started to pay attention to the potential of Docker.
What is your favorite Docker command?
I really enjoy “docker diff”. I find it incredibly useful to see what files/directories are being added or modified by the process within the container. It’s great for debugging. A close second would be “docker stats”.
What is your top tip for working with Docker that others may not know?
When possible, own your own base image and pin tooling versions. It’s convenient to use the public images on Docker Hub but when working in an organization where hundreds of developers are relying on the same base image try to bring it in house. We set up an image “bakery” at Demonware where we would build our base images nightly, with pinned versions of tools included, ran extensive tests, triggered downstream pipelines and verified that our base image was always in a good state. From experience it’s the base image where the most “bloat” occurs and keeping it lean can also help when moving the image around your infrastructure.
What’s the coolest Docker demo you have done/seen ?
My favorite demo was by Casey Bisson from Joyent. In the demo he showed off how Triton worked and how he could scale from a single container running locally out to an entire datacenter by just updating a single endpoint. This was in 2016 and I still love the simplicity of this approach.
What have you worked on in the past six months that you’re particularly proud of?
I’ve been using containers daily since late 2013 and still find new uses for them regularly. I’m currently working on improving developer tooling and UX at DIGIT Games Studios in Dublin. Part of this work includes containerizing our tooling and making it accessible in multiple ways including from Slack, command line, callable by API and even runnable from a spreadsheet. Docker enables us to bring the tooling closer to the end user whether they are technical or non technical. Rolling out updates to tools is now trivial.
What do you anticipate will be Docker’s biggest announcement this year?
Development Environments (DE) has received a huge amount of positive feedback based on the early announcements. I think the potential of DE is huge. To have these environments tightly integrated into a developers IDE, easily shareable and customizable will remove existing friction and help developers move from idea to production with greater velocity.
What are some personal goals for the next year with respect to the Docker community?
Our last Docker Dublin Meetup was in February 2020 and with a community of over 3000 members I’d like to start the in-person meetups again in 2022. I’d also like to continue running more Docker Workshops around Ireland and take the power of containerization out to other communities.
What talk would you most love to see at DockerCon 2022?
Any talks that include gaming would be great. I particularly loved the live migration demo of Quake a few years ago. Some studios are doing really cool stuff with containers. As early adopters of containers, Demonware may have some useful experiences to share with regard to their journey from self-hosted Kubernetes to the cloud.
Looking to the distant future, what is the technology that you’re most excited about and that you think holds a lot of promise?
That’s a tough question. Any technology that focuses on the user experience first and is a joy to use excites me. From a gaming perspective Augmented Reality has so much potential, particularly for mobile gaming. I’m not going to mention Blockchain or NFTs since I don’t know enough about either to comment. They may have their place but if they negatively impact the environment as suggested then perhaps they need to go back to the drawing board for a few more iterations.
Rapid fire questions…
What new skill have you mastered during the pandemic?
Mindfulness.
Cats or Dogs?
Dogs.
Salty, sour or sweet?
Sweet.
Beach or mountains?
Mountains.
Your most often used emoji?
DockerCon2022
Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/
The post Docker Captain Take 5 – Thomas Shaw appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
How to Purchase a Docker Subscription from a Reseller
With the grace period for the new Docker subscription service agreement ending very soon on January 31, 2022, we want to make it easy for customers to use their preferred reseller to purchase a Docker subscription.
That’s why we recently announced that your preferred reseller can now purchase a Docker Business subscription through Nuaware. That’s right, Docker and Nuaware have teamed up to advance the adoption of Docker throughout the software engineering departments of large organizations. Nuaware has established relationships with 1,000’s of resellers around the globe.
Nuaware is a Trusted Partner
Nuaware is a specialized distributor in developer tools, DevOps, Cloud, and Cloud-Native technologies. They help large enterprises adopt modern architectures by supporting them with the right products, training, and partner ecosystem. Nuaware is part of Exclusive Networks, a global provider of cybersecurity and cloud solutions with offices in over 50 countries across 5 continents with a reseller network exceeding 18,000 partners. Nuaware was founded with the view that new platform and software development technologies are an ecosystem business and in order for an enterprise to successfully adopt new technologies like microservice architectures in production, they need specialist partners in many areas. Nuaware helps select and introduce the right technologies, partners, and training to take ideas into production.
Serving Our Customers Around the Globe
Developers using Docker are more productive, build more secure software, and are able to collaborate more effectively. Combined with Nuaware’s developer tool expertise and the reach of Exclusive Networks’ channel ecosystem of 18,000 specialist channel partners, Docker has an unparalleled ability to service our clients’ needs around the globe. No matter how you prefer or need to purchase your software, Docker has you covered.
Get Started Today
Learn more about Docker Business and get started today by visiting https://www.nuaware.com/docker
The post How to Purchase a Docker Subscription from a Reseller appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
