Kategorie: Kubernetes

Extensible Admission is Beta

In this post we review a feature, available in the Kubernetes API server, that allows you to implement arbitrary control decisions and which has matured considerably in Kubernetes 1.9.The admission stage of API server processing is one of the most powerful tools for securing a Kubernetes cluster by restricting the objects that can be created, but it has always been limited to compiled code. In 1.9, we promoted webhooks for admission to beta, allowing you to leverage admission from outside the API server process.What is Admission?Admission is the phase of handling an API server request that happens before a resource is persisted, but after authorization. Admission gets access to the same information as authorization (user, URL, etc) and the complete body of an API request (for most requests).The admission phase is composed of individual plugins, each of which are narrowly focused and have semantic knowledge of what they are inspecting. Examples include: PodNodeSelector (influences scheduling decisions), PodSecurityPolicy (prevents escalating containers), and ResourceQuota (enforces resource allocation per namespace).Admission is split into two phases:Mutation, which allows modification of the body content itself as well as rejection of an API request.Validation, which allows introspection queries and rejection of an API request.An admission plugin can be in both phases, but all mutation happens before validation.MutationThe mutation phase of admission allows modification of the resource content before it is persisted. Because the same field can be mutated multiple times while in the admission chain, the order of the admission plugins in the mutation matters.One example of a mutating admission plugin is the `PodNodeSelector` plugin, which uses an annotation on a namespace `namespace.annotations[“scheduler.alpha.kubernetes.io/node-selector”]` to find a label selector and add it to the `pod.spec.nodeselector` field. This positively restricts which nodes the pods in a particular namespace can land on, as opposed to taints, which provide negative restriction (also with an admission plugin).ValidationThe validation phase of admission allows the enforcement of invariants on particular API resources. The validation phase runs after all mutators finish to ensure that the resource isn’t going to change again.One example of a validation admission plugin is also the `PodNodeSelector` plugin, which ensures that all pods’ `spec.nodeSelector` fields are constrained by the node selector restrictions on the namespace. Even if a mutating admission plugin tries to change the `spec.nodeSelector` field after the PodNodeSelector runs in the mutating chain, the PodNodeSelector in the validating chain prevents the API resource from being created because it fails validation.What are admission webhooks?Admission webhooks allow a Kubernetes installer or a cluster-admin to add mutating and validating admission plugins to the admission chain of `kube-apiserver` as well as any extensions apiserver based on k8s.io/apiserver 1.9, like metrics, service-catalog, or kube-projects, without recompiling them. Both kinds of admission webhooks run at the end of their respective chains and have the same powers and limitations as compiled admission plugins.What are they good for?Webhook admission plugins allow for mutation and validation of any resource on any API server, so the possible applications are vast. Some common use-cases include:Mutation of resources like pods. Istio has talked about doing this to inject side-car containers into pods. You could also write a plugin which forcefully resolves image tags into image SHAs.Name restrictions. On multi-tenant systems, reserving namespaces has emerged as a use-case.Complex CustomResource validation. Because the entire object is visible, a clever admission plugin can perform complex validation on dependent fields (A requires B) and even external resources (compare to LimitRanges).Security response. If you forced image tags into image SHAs, you could write an admission plugin that prevents certain SHAs from running.RegistrationWebhook admission plugins of both types are registered in the API, and all API servers (kube-apiserver and all extension API servers) share a common config for them. During the registration process, a webhook admission plugin describes:How to connect to the webhook admission serverHow to verify the webhook admission server (Is it really the server I expect?)Where to send the data at that server (which URL path)Which resources and which HTTP verbs it will handleWhat an API server should do on connection failures (for example, if the admission webhook server goes down)1 apiVersion: admissionregistration.k8s.io/v1beta12 kind: ValidatingWebhookConfiguration3 metadata:4   name: namespacereservations.admission.online.openshift.io5 webhooks:6 – name: namespacereservations.admission.online.openshift.io7   clientConfig:8     service:9       namespace: default10      name: kubernetes11     path: /apis/admission.online.openshift.io/v1alpha1/namespacereservations12    caBundle: KUBE_CA_HERE13  rules:14  – operations:15    – CREATE16    apiGroups:17    – “”18    apiVersions:19    – “*”20    resources:21    – namespaces22  failurePolicy: FailLine 6: `name` – the name for the webhook itself. For mutating webhooks, these are sorted to provide ordering.Line 7: `clientConfig` – provides information about how to connect to, trust, and send data to the webhook admission server.Line 13: `rules` – describe when an API server should call this admission plugin. In this case, only for creates of namespaces. You can specify any resource here so specifying creates of `serviceinstances.servicecatalog.k8s.io` is also legal.Line 22: `failurePolicy` – says what to do if the webhook admission server is unavailable. Choices are “Ignore” (fail open) or “Fail” (fail closed). Failing open makes for unpredictable behavior for all clients.Authentication and trustBecause webhook admission plugins have a lot of power (remember, they get to see the API resource content of any request sent to them and might modify them for mutating plugins), it is important to consider:How individual API servers verify their connection to the webhook admission server How the webhook admission server authenticates precisely which API server is contacting itWhether that particular API server has authorization to make the requestThere are three major categories of connection:From kube-apiserver or extension-apiservers to externally hosted admission webhooks (webhooks not hosted in the cluster)From kube-apiserver to self-hosted admission webhooksFrom extension-apiservers to self-hosted admission webhooksTo support these categories, the webhook admission plugins accept a kubeconfig file which describes how to connect to individual servers. For interacting with externally hosted admission webhooks, there is really no alternative to configuring that file manually since the authentication/authorization and access paths are owned by the server you’re hooking to.For the self-hosted category, a cleverly built webhook admission server and topology can take advantage of the safe defaulting built into the admission plugin and have a secure, portable, zero-config topology that works from any API server.Simple, secure, portable, zero-config topologyIf you build your webhook admission server to also be an extension API server, it becomes possible to aggregate it as a normal API server. This has a number of advantages:Your webhook becomes available like any other API under default kube-apiserver service `kubernetes.default.svc` (e.g. https://kubernetes.default.svc/apis/admission.example.com/v1/mymutatingadmissionreviews). Among other benefits, you can test using `kubectl`.Your webhook automatically (without any config) makes use of the in-cluster authentication and authorization provided by kube-apiserver. You can restrict access to your webhook with normal RBAC rules.Your extension API servers and kube-apiserver automatically (without any config) make use of their in-cluster credentials to communicate with the webhook.Extension API servers do not leak their service account token to your webhook because they go through kube-apiserver, which is a secure front proxy.Source: https://drive.google.com/a/redhat.com/file/d/12nC9S2fWCbeX_P8nrmL6NgOSIha4HDNp In short: a secure topology makes use of all security mechanisms of API server aggregation and additionally requires no additional configuration.Other topologies are possible but require additional manual configuration as well as a lot of effort to create a secure setup, especially when extension API servers like service catalog come into play. The topology above is zero-config and portable to every Kubernetes cluster.How do I write a webhook admission server?Writing a full server complete with authentication and authorization can be intimidating. To make it easier, there are projects based on Kubernetes 1.9 that provide a library for building your webhook admission server in 200 lines or less. Take a look at the generic-admission-apiserver and the kubernetes-namespace-reservation projects for the library and an example of how to build your own secure and portable webhook admission server.With the admission webhooks introduced in 1.9 we’ve made Kubernetes even more adaptable to your needs. We hope this work, driven by both Red Hat and Google, will enable many more workloads and support ecosystem components. (Istio is one example.) Now is a good time to give it a try! If you’re interested in giving feedback or contributing to this area, join us in the SIG API machinery.
Quelle: kubernetes

Using Your Own Private Registry with Docker Enterprise Edition

One of the things that makes Docker really cool, particularly compared to using virtual machines, is how easy it is to move around Docker images. If you’ve already been using Docker, you’ve almost certainly pulled images from Docker Hub. Docker Hub is Docker’s cloud-based registry service and has tens of thousands of Docker images to choose from. If you’re developing your own software and creating your own Docker images though, you’ll want your own private Docker registry. This is particularly true if you have images with proprietary licenses, or if you have a complex continuous integration (CI) process for your build system.
Docker Enterprise Edition includes Docker Trusted Registry (DTR), a highly available registry with secure image management capabilities which was built to run either inside of your own data center or on your own cloud-based infrastructure. In the next few weeks, we’ll go over how DTR is a critical component of delivering a secure, repeatable and consistent software supply chain.  You can get started with it today through our free hosted demo or by downloading and installing the free 30-day trial. The steps to get started with your own installation are below.
Setting Up Docker Enterprise Edition
Docker Trusted Registry runs on top of Universal Control Plane (UCP), so to begin let’s install a single-node cluster. If you’ve already got your own UCP cluster, you can skip this step.  On your docker host, run the command:
  # Pull and install UCP
  docker run -it –rm -v /var/run/docker.sock:/var/run/docker.sock –name ucp docker/ucp:latest install
Once UCP is up and running, there are a few more things you should do before you install DTR. Open up your browser against the UCP instance you just installed. There should be a link to it at the end of your log output. If you have already have a Docker Enterprise Edition license, go ahead and upload it through the UI. If you don’t, visit the Docker Store and pick up a free, 30-day trial.
Once you’ve got licensing squared away, you’re probably going to want to change the port which UCP is running on. Since this is a single node cluster, DTR and UCP are going to want to use the same TCP ports for running their web services. If you’ve got a UCP swarm with more than one node, this probably isn’t a problem because DTR will look for a node which has the required free ports. Inside of UCP, click on Admin Settings -> Cluster Configuration and change the Controller Port to something like 5443.
Installing DTR
We’re going to install a simple, single-node instance of Docker Trusted Registry.  If you were setting up your DTR for production use, you would likely set things up in High Availability (HA) mode which would require a different type of storage such as a cloud-based object store, or NFS. Since this is a single-node instance, we’re going to stick with the default local storage.
First we need to pull the DTR bootstrap image. The bootstrap image is a tiny, self-contained installer which connects to UCP and sets up all of the containers, volumes, and logical networks required to get DTR up and running.
Use the command:
  # Pull and run the DTR bootstrapper
  docker run -it –rm docker/dtr:latest install –ucp-insecure-tls
NOTE:  Both UCP and DTR by default come with their own certs which won’t be recognized by your system.  If you’ve set up UCP with TLS certs which are trusted by your system, you can omit the `–ucp-insecure-tls` option. Alternatively, you can use the `–ucp-ca` option which will let you specify the UCP CA certificate directly.
The DTR bootstrap image should then ask you for a couple of settings, such as the URL of your UCP installation and your UCP admin username and password.  It should only take a minute or two to pull all of the DTR images and set everything up.
Keeping Everything Secure
Once everything is up and running, you’re ready to push and pull images to and from
the registry.  Before we do that step though, let’s set up our TLS certificates so that we can securely talk to DTR.
On Linux, we can use these commands (just make certain you change DTR_HOSTNAME to reflect the DTR we just set up):
  # Pull the CA certificate from DTR (you can use wget if curl is unavailable)
  DTR_HOSTNAME=<Your DTR hostname>
  curl -k https://$(DTR_HOSTNAME)/ca > $(DTR_HOSTNAME).crt
  sudo mkdir /etc/docker/certs.d/$(DTR_HOSTNAME)
  sudo cp $(DTR_HOSTNAME) /etc/docker/certs.d/$(DTR_HOSTNAME)
  # Restart the docker daemon (use `sudo service docker restart` on Ubuntu 14.04)
  sudo systemctl restart docker
On Docker for Mac and Windows, we’ll set up our client a little bit differently.  Go in to Settings -> Daemon and in the Insecure Registries section, enter in your DTR hostname.  Click Apply, and your docker daemon should restart and you should be good to go.
Pushing and Pulling Images
We now need to set up a repository to hold an image. This is a little bit different than Docker Hub which automatically creates a repository if one doesn’t exist when you do a
docker push. To create the repository, point your browser to https://<Your DTR hostname> and
then sign-in with your admin credentials when prompted. If you added a license to UCP, that
license will automatically have been picked up by DTR. If not, make certain you upload
your license now.
Once you’re in, click on the ‘New Repository` button and create a new repository.
We’ll create a repo to hold Alpine linux, so type `alpine` in the name field, and click
`Save` (it’s labelled `Create` in DTR 2.5 and newer).
Now let’s go back to our shell and type the commands:
  # Pull the latest version of Alpine Linux
  docker pull alpine:latest
  # Sign in to your new DTR instance
  docker login <Your DTR hostname>
  # Tag Alpine to be able to push it to your DTR
  docker tag alpine:latest <Your DTR hostname>/admin/alpine:latest
  # Push the image to DTR
  docker push <Your DTR hostname>/admin/alpine:latest
And that’s it!  We just pulled a copy of the latest Alpine Linux, re-tagged it so that we could store it inside of DTR, and then pushed it to our private registry.  If you want to pull that image to a different Docker engine, set up your DTR certs as shown above, and issue the command:
   # Pull the image from DTR
   docker pull <Your DTR hostname>/admin/alpine:latest
DTR has a lot of great image management features built right in such as image caching, mirroring, scanning, signing, and even automated supply chain policies.  We’ll leave these to future blog posts which we can explore in more detail.

Step-by-step instructions on how to setup and use your own private registry with #Docker Enterprise…Click To Tweet

To learn more about Docker Enterprise Edition:

Visit the website and view pricing
Read more about Docker EE customers and the benefits they’re seeing
Don’t have time to install and configure Docker EE? Register for the free hosted trial to test drive Docker EE in just a few minutes

The post Using Your Own Private Registry with Docker Enterprise Edition appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Docker for Mac with Kubernetes

You heard about it at DockerCon Europe and now it is here: we are proud to announce that Docker for Mac with beta Kubernetes support is now publicly available as part of the Edge release channel. We hope you are as excited as we are!
With this release you can now run a single node Kubernetes cluster right on your Mac and use both kubectl commands and docker commands to control your containers.
First, a few things to keep in mind:

Docker for Mac required
Kubernetes features are only accessible on macOS for now; Docker for Windows and Docker Enterprise Edition betas will follow at a later date. If you need to install a new copy of Docker for Mac you can download it from the Docker Store.
Edge channel required
Kubernetes support is still considered experimental with this release, so to enable the download and use of Kubernetes components you must be on the Edge channel. The Docker for Mac version should be 17.12.0-ce-mac45 or later after updating.
Already using other Kubernetes tools?
If you are already running a version of kubectl pointed at another environment, for example minikube, you will want to follow the activation instructions to change contexts to docker-for-desktop.

What You Can Do
Docker for Mac and Windows are the most popular way to configure a Docker dev environment and are used everyday by hundreds of thousands of developers to build, test and debug containerized apps. Developers building both docker-compose and Swarm-based apps, and apps destined for deployment on Kubernetes can now get a simple-to-use development system that takes optimal advantage of their laptop or workstation. All container tasks – build, run and push – will run on the same Docker instance with a shared set of images, volumes and containers. Docker for Mac is simple to install, so you can have Docker containers running on your Mac in just a few minutes. And Docker for Mac auto-updates so you continue getting the latest Docker product revisions.
With experimental Kubernetes support in Docker CE for Mac, Docker can provide users an end-to-end suite of container-management software and services that span from developer workstations running Docker for Mac or Windows, through test and CI/CD, through to production systems on-premises or in the cloud running Docker Enterprise Edition (EE).
The beauty of building with Docker for Mac or Windows is that you can deploy the exact same set of Docker container images on your desktop as you do with Docker Enterprise Edition (EE) on your production systems. Docker for Mac or Windows is a single node system for building, testing and preparing to ship applications; Docker EE provides the security, control, and scale needed to manage your production applications. You eliminate the “it worked on my machine” problem because you have the same Docker containers running on the same Docker engines, and the same Docker Swarm and Kubernetes orchestrators (coming soon to EE).

Things To Try
If you are new to Kubernetes and looking for some introductory exercises to try, here are a few resources:

The Docker for Mac Kubernetes page has instructions for getting an example app up and running
Follow along with Docker Developer Advocate Elton Stoneman during his short video, demonstrating activating Kubernetes and deploying an application using both Docker compose and a Kubernetes manifest.

Send Us Your Feedback
Send us your feedback, ideas for improvement, bugs, complaints and more so we can make Docker Desktop better. You can use the Docker community forums for general discussions and you can also directly file technical issues on Github.

Docker for Mac with Experimental Kubernetes support is here! #dockerformac #docker #kubernetesClick To Tweet

Call to Action

Get Docker for Mac
Activate experimental Kubernetes support in Docker for Mac
Watch our DockerCon Europe 2017 Kubernetes Announcement

The post Docker for Mac with Kubernetes appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

PaddlePaddle Fluid: Elastic Deep Learning on Kubernetes

Editor’s note: Today’s post is a joint post from the deep learning team at Baidu and the etcd team at CoreOS.PaddlePaddle Fluid: Elastic Deep Learning on KubernetesTwo open source communities—PaddlePaddle, the deep learning framework originated in Baidu, and Kubernetes®, the most famous containerized application scheduler—are announcing the Elastic Deep Learning (EDL) feature in PaddlePaddle’s new release codenamed Fluid.Fluid EDL includes a Kubernetes controller, PaddlePaddle auto-scaler, which changes the number of processes of distributed jobs according to the idle hardware resource in the cluster, and a new fault-tolerable architecture as described in the PaddlePaddle design doc.Industrial deep learning requires significant computation power. Research labs and companies often build GPU clusters managed by SLURM, MPI, or SGE. These clusters either run a submitted job if it requires less than the idle resource, or pend the job for an unpredictably long time. This approach has its drawbacks: in an example with 99 available nodes and a submitted job that requires 100, the job has to wait without using any of the available nodes. Fluid works with Kubernetes to power elastic deep learning jobs, which often lack optimal resources, by helping to expose potential algorithmic problems as early as possible.Another challenge is that industrial users tend to run deep learning jobs as a subset stage of the complete data pipeline, including the web server and log collector. Such general-purpose clusters require priority-based elastic scheduling. This makes it possible to run more processes in the web server job and less in deep learning during periods of high web traffic, then prioritize deep learning when web traffic is low. Fluid talks to Kubernetes’ API server to understand the global picture and orchestrate the number of processes affiliated with various jobs.In both scenarios, PaddlePaddle jobs are tolerant to a process spikes and decreases. We achieved this by implementing the new design, which introduces a master process in addition to the old PaddlePaddle architecture as described in a previous blog post. In the new design, as long as there are three processes left in a job, it continues. In extreme cases where all processes are killed, the job can be restored and resume.We tested Fluid EDL for two use cases: 1) the Kubernetes cluster runs only PaddlePaddle jobs; and 2) the cluster runs PaddlePaddle and Nginx jobs.In the first test, we started up to 20 PaddlePaddle jobs one by one with a 10-second interval. Each job has 60 trainers and 10 parameter server processes, and will last for hours. We repeated the experiment 20 times: 10 with FluidEDL turned off and 10 with FluidEDL turned on. In Figure one, solid lines correspond to the first 10 experiments and dotted lines the rest. In the upper part of the figure, we see that the number of pending jobs increments monotonically without EDL. However, when EDL is turned on, resources are evenly distributed to all jobs. Fluid EDL kills some existing processes to make room for new jobs and jobs coming in at a later point in time. In both cases, the cluster is equally utilized (see lower part of figure).Figure 1. Fluid EDL evenly distributes resource among jobs.In the second test, each experiment ran 400 Nginx pods, which has higher priority than the six PaddlePaddle jobs. Initially, each PaddlePaddle job had 15 trainers and 10 parameter servers. We killed 100 Nginx pods every 90 seconds until 100 left, and then we started to increase the number of Nginx jobs by 100 every 90 seconds. The upper part of Figure 2 shows this process. The middle of the diagram shows that Fluid EDL automatically started some PaddlePaddle processes by decreasing Nginx pods, and killed PaddlePaddle processes by increasing Nginx pods later on. As a result, the cluster maintains around 90% utilization as shown in the bottom of the figure. When Fluid EDL was turned off, there were no PaddlePaddle processes autoincrement, and the utilization fluctuated with the varying number of Nginx pods.Figure 2. Fluid changes PaddlePaddle processes with the change of Nginx processes.We continue to work on FluidEDL and welcome comments and contributions. Visit the PaddlePaddle repo, where you can find the design doc, a simple tutorial, and experiment details.Xu Yan (Baidu Research)Helin Wang (Baidu Research)Yi Wu (Baidu Research)Xi Chen (Baidu Research)Weibao Gong (Baidu Research)Xiang Li (CoreOS)- Yi Wang (Baidu Research)
Quelle: kubernetes

Using eBPF in Kubernetes

IntroductionKubernetes provides a high-level API and a set of components that hides almost all of the intricate and—to some of us—interesting details of what happens at the systems level. Application developers are not required to have knowledge of the machines’ IP tables, cgroups, namespaces, seccomp, or, nowadays, even the container runtime that their application runs on top of. But underneath, Kubernetes and the technologies upon which it relies (for example, the container runtime) heavily leverage core Linux functionalities.This article focuses on a core Linux functionality increasingly used in networking, security and auditing, and tracing and monitoring tools. This functionality is called extended Berkeley Packet Filter (eBPF)Note: In this article we use both acronyms: eBPF and BPF. The former is used for the extended BPF functionality, and the latter for “classic” BPF functionality. What is BPF?BPF is a mini-VM residing in the Linux kernel that runs BPF programs. Before running, BPF programs are loaded with the bpf() syscall and are validated for safety: checking for loops, code size, etc. BPF programs are attached to kernel objects and executed when events happen on those objects—for example, when a network interface emits a packet.BPF SuperpowersBPF programs are event-driven by definition, an incredibly powerful concept, and executes code in the kernel when an event occurs. Netflix’s Brendan Gregg refers to BPF as a Linux superpower.The ‘e’ in eBPFTraditionally, BPF could only be attached to sockets for socket filtering. BPF’s first use case was in `tcpdump`. When you run `tcpdump` the filter is compiled into a BPF program and attached to a raw `AF_PACKET` socket in order to print out filtered packets.But over the years, eBPF added the ability to attach to other kernel objects. In addition to socket filtering, some supported attach points are:Kprobes (and userspace equivalents uprobes)TracepointsNetwork schedulers or qdiscs for classification or action (tc)XDP (eXpress Data Path)This and other, newer features like in-kernel helper functions and shared data-structures (maps) that can be used to communicate with user space, extend BPF’s capabilities.Existing Use Cases of eBPF with KubernetesSeveral open-source Kubernetes tools already use eBPF and many use cases warrant a closer look, especially in areas such as networking, monitoring and security tools.Dynamic Network Control and Visibility with CiliumCilium is a networking project that makes heavy use of eBPF superpowers to route and filter network traffic for container-based systems. By using eBPF, Cilium can dynamically generate and apply rules—even at the device level with XDP—without making changes to the Linux kernel itself.The Cilium Agent runs on each host. Instead of managing IP tables, it translates network policy definitions to BPF programs that are loaded into the kernel and attached to a container’s virtual ethernet device. These programs are executed—rules applied—on each packet that is sent or received.This diagram shows how the Cilium project works:Depending on what network rules are applied, BPF programs may be attached with tc or XDP. By using XDP, Cilium can attach the BPF programs at the lowest possible point, which is also the most performant point in the networking software stack.If you’d like to learn more about how Cilium uses eBPF, take a look at the project’s BPF and XDP reference guide.Tracking TCP Connections in Weave ScopeWeave Scope is a tool for monitoring, visualizing and interacting with container-based systems. For our purposes, we’ll focus on how Weave Scope gets the TCP connections.Weave Scope employs an agent that runs on each node of a cluster. The agent monitors the system, generates a report and sends it to the app server. The app server compiles the reports it receives and presents the results in the Weave Scope UI.To accurately draw connections between containers, the agent attaches a BPF program to kprobes that track socket events: opening and closing connections. The BPF program, tcptracer-bpf, is compiled into an ELF object file and loaded using gopbf.(As a side note, Weave Scope also has a plugin that make use of eBPF: HTTP statistics.)To learn more about how this works and why it’s done this way, read this extensive post that the Kinvolk team wrote for the Weaveworks Blog. You can also watch a recent talk about the topic.Limiting syscalls with seccomp-bpfLinux has more than 300 system calls (read, write, open, close, etc.) available for use—or misuse. Most applications only need a small subset of syscalls to function properly. seccomp is a Linux security facility used to limit the set of syscalls that an application can use, thereby limiting potential misuse.The original implementation of seccomp was highly restrictive. Once applied, if an application attempted to do anything beyond reading and writing to files it had already opened, seccomp sent a `SIGKILL` signal.seccomp-bpf enables more complex filters and a wider range of actions. Seccomp-bpf, also known as seccomp mode 2, allows for applying custom filters in the form of BPF programs. When the BPF program is loaded, the filter is applied to each syscall and the appropriate action is taken (Allow, Kill, Trap, etc.).seccomp-bpf is widely used in Kubernetes tools and exposed in Kubernetes itself. For example, seccomp-bpf is used in Docker to apply custom seccomp security profiles, in rkt to apply seccomp isolators, and in Kubernetes itself in its Security Context.But in all of these cases the use of BPF is hidden behind libseccomp. Behind the scenes, libseccomp generates BPF code from rules provided to it. Once generated, the BPF program is loaded and the rules applied.Potential Use Cases for eBPF with KuberneteseBPF is a relatively new Linux technology. As such, there are many uses that remain unexplored. eBPF itself is also evolving: new features are being added in eBPF that will enable new use cases that aren’t currently possible. In the following sections, we’re going to look at some of these that have only recently become possible and ones on the horizon. Our hope is that these features will be leveraged by open source tooling.Pod and container level network statisticsBPF socket filtering is nothing new, but BPF socket filtering per cgroup is. Introduced in Linux 4.10, cgroup-bpf allows attaching eBPF programs to cgroups. Once attached, the program is executed for all packets entering or exiting any process in the cgroup.A cgroup is, amongst other things, a hierarchical grouping of processes. In Kubernetes, this grouping is found at the container level. One idea for making use of cgroup-bpf, is to install BPF programs that collect detailed per-pod and/or per-container network statistics.Generally, such statistics are collected by periodically checking the relevant file in Linux’s `/sys` directory or using Netlink. By using BPF programs attached to cgroups for this, we can get much more detailed statistics: for example, how many packets/bytes on tcp port 443, or how many packets/bytes from IP 10.2.3.4. In general, because BPF programs have a kernel context, they can safely and efficiently deliver more detailed information to user space.To explore the idea, the Kinvolk team implemented a proof-of-concept: https://github.com/kinvolk/cgnet. This project attaches a BPF program to each cgroup and exports the information to Prometheus.There are of course other interesting possibilities, like doing actual packet filtering. But the obstacle currently standing in the way of this is having cgroup v2 support—required by cgroup-bpf—in Docker and Kubernetes.Application-applied LSMLinux Security Modules (LSM) implements a generic framework for security policies in the Linux kernel. SELinux and AppArmor are examples of these. Both of these implement rules at a system-global scope, placing the onus on the administrator to configure the security policies.Landlock is another LSM under development that would co-exist with SELinux and AppArmor. An initial patchset has been submitted to the Linux kernel and is in an early stage of development. The main difference with other LSMs is that Landlock is designed to allow unprivileged applications to build their own sandbox, effectively restricting themselves instead of using a global configuration. With Landlock, an application can load a BPF program and have it executed when the process performs a specific action. For example, when the application opens a file with the open() system call, the kernel will execute the BPF program, and, depending on what the BPF program returns, the action will be accepted or denied.In some ways, it is similar to seccomp-bpf: using a BPF program, seccomp-bpf allows unprivileged processes to restrict what system calls they can perform. Landlock will be more powerful and provide more flexibility. Consider the following system call:Cfd = open(“myfile.txt”, O_RDWR);The first argument is a “char *”, a pointer to a memory address, such as `0xab004718`. With seccomp, a BPF program only has access to the parameters of the syscall but cannot dereference the pointers, making it impossible to make security decisions based on a file. seccomp also uses classic BPF, meaning it cannot make use of eBPF maps, the mechanism for interfacing with user space. This restriction means security policies cannot be changed in seccomp-bpf based on a configuration in an eBPF map.BPF programs with Landlock don’t receive the arguments of the syscalls but a reference to a kernel object. In the example above, this means it will have a reference to the file, so it does not need to dereference a pointer, consider relative paths, or perform chroots.Use Case: Landlock in Kubernetes-based serverless frameworksIn Kubernetes, the unit of deployment is a pod. Pods and containers are the main unit of isolation. In serverless frameworks, however, the main unit of deployment is a function. Ideally, the unit of deployment equals the unit of isolation. This puts serverless frameworks like Kubeless or OpenFaaS into a predicament: optimize for unit of isolation or deployment?To achieve the best possible isolation, each function call would have to happen in its own container—ut what’s good for isolation is not always good for performance. Inversely, if we run function calls within the same container, we increase the likelihood of collisions.By using Landlock, we could isolate function calls from each other within the same container, making a temporary file created by one function call inaccessible to the next function call, for example. Integration between Landlock and technologies like Kubernetes-based serverless frameworks would be a ripe area for further exploration.Auditing kubectl-exec with eBPFIn Kubernetes 1.7 the audit proposal started making its way in. It’s currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster. While these events log Kubernetes events, they don’t currently provide the level of visibility that some may require. For example, while we can see that someone has used `kubectl exec` to enter a container, we are not able to see what commands were executed in that session. With eBPF one can attach a BPF program that would record any commands executed in the `kubectl exec` session and pass those commands to a user-space program that logs those events. We could then play that session back and know the exact sequence of events that took place.Learn more about eBPFIf you’re interested in learning more about eBPF, here are some resources:A comprehensive reading list about eBPF for doing just thatBCC (BPF Compiler Collection) provides tools for working with eBPF as well as many example tools making use of BCC.Some videosBPF: Tracing and More by Brendan GreggCilium – Container Security and Networking Using BPF and XDP by Thomas GrafUsing BPF in Kubernetes by Alban CrequyConclusionWe are just starting to see the Linux superpowers of eBPF being put to use in Kubernetes tools and technologies. We will undoubtedly see increased use of eBPF. What we have highlighted here is just a taste of what you might expect in the future. What will be really exciting is seeing how these technologies will be used in ways that we have not yet thought about. Stay tuned!The Kinvolk team will be hanging out at the Kinvolk booth at KubeCon in Austin. Come by to talk to us about all things, Kubernetes, Linux, container runtimes and yeah, eBPF.
Quelle: kubernetes