Kategorie: Kubernetes

Visualize Kubelet Performance with Node Dashboard

In Kubernetes 1.4, we introduced a new node performance analysis tool, called the node performance dashboard, to visualize and explore the behavior of the Kubelet in much richer details. This new feature will make it easy to understand and improve code performance for Kubelet developers, and lets cluster maintainer to decide configurations according to provided Service Level Objectives (SLOs).BackgroundA Kubernetes cluster is made up of both master and worker nodes. The master node manages the cluster’s state, and the worker nodes do the actual work of running and managing pods. To do so, on each worker node, a binary, called Kubelet, watches for any changes in pod configuration, and takes corresponding actions to make sure that containers run successfully. High performance of the Kubelet, such as low latency to converge with new pod configuration and efficient housekeeping with low resource usage, is essential for the entire Kubernetes cluster. To measure this performance, Kubernetes uses end-to-end (e2e) tests to continuously monitor benchmark changes of latest builds with new features.Kubernetes SLOs are defined by the following benchmarks:     * API responsiveness: 99% of all API calls return in less than 1s.     * Pod startup time: 99% of pods and their containers (with pre-pulled images) start within 5s.Prior to 1.4 release, we’ve only measured and defined these at the cluster level, opening up the risk that other factors could influence the results. Beyond these, we also want to have more performance related SLOs such as the maximum number of pods for a specific machine type allowing maximum utilization of your cluster. In order to do the measurement correctly, we want to introduce a set of tests isolated to just a node’s performance. In addition, we aim to collect more fine-grained resource usage and operation tracing data of Kubelet from the new tests.Data CollectionThe node specific density and resource usage tests are now added into e2e-node test set since 1.4. The resource usage is measured by a standalone cAdvisor pod for flexible monitoring interval (comparing with Kubelet integrated cAdvisor). The performance data, such as latency and resource usage percentile, are recorded in persistent test result logs. The tests also record time series data such as creation time, running time of pods, as well as real-time resource usage. Tracing data of Kubelet operations are recorded in its log stored together with test results.Node Performance DashboardSince Kubernetes 1.4, we are continuously building the newest Kubelet code and running node performance tests. The data is collected by our new performance dashboard available at node-perf-dash.k8s.io. Figure 1 gives a preview of the dashboard. You can start to explore it by selecting a test, either using the drop-down list of short test names (region (a)) or by choosing test options one by one (region (b)). The test details show up in region (c) containing the full test name from Ginkgo (the Go test framework used by Kubernetes). Then select a node type (image and machine) in region (d).Figure 1. Select a test to display in node performance dashboard.The “BUILDS” page exhibits the performance data across different builds (Figure 2). The plots include pod startup latency, pod creation throughput, and CPU/memory usage of Kubelet and runtime (currently Docker). In this way it’s easy to monitor the performance change over time as new features are checked in.Figure 2. Performance data across different builds.Compare Different Node ConfigurationsIt’s always interesting to compare the performance between different configurations, such as comparing startup latency of different machine types, different numbers of pods, or comparing resource usage of hosting different number of pods. The dashboard provides a convenient way to do this. Just click the “Compare it” button the right up corner of test selection menu (region (e) in Figure 1). The selected tests will be added to a comparison list in the “COMPARISON” page, as shown in Figure 3. Data across a series of builds are aggregated to a single value to facilitate comparison and are displayed in bar charts.Figure 3. Compare different test configurations.Time Series and Tracing: Diving Into Performance DataPod startup latency is an important metric for Kubelet, especially when creating a large number of pods per node. Using the dashboard you can see the change of latency, for example, when creating 105 pods, as shown in Figure 4. When you see the highly variable lines, you might expect that the variance is due to different builds. However, as these test here were run against the same Kubernetes code, we can conclude the variance is due to performance fluctuation. The variance is close to 40s when we compare the 99% latency of build #162 and #173, which is very large. To drill into the source of the fluctuation, let’s check out the “TIME SERIES” page.Figure 4. Pod startup latency when creating 105 pods.Looking specifically at build 162, we are able to see that the tracing data plotted in the pod creation latency chart (Figure 5). Each curve is an accumulated histogram of the number of pod operations which have already arrive at a certain tracing probe. The timestamp of tracing pod is either collected from the performance tests or by parsing the Kubelet log. Currently we collect the following tracing data:”create” (in test): the test creates pods through API client;”running” (in test): the test watches that pods are running from API server;”pod_config_change”: pod config change detected by Kubelet SyncLoop;”runtime_manager”: runtime manager starts to create containers;”infra_container_start”: the infra container of a pod starts;”container_start': the container of a pod starts;”pod_running”: a pod is running;”pod_status_running”: status manager updates status for a running pod;The time series chart illustrates that it is taking a long time for the status manager to update pod status (the data of “running” is not shown since it overlaps with “pod_status_running”). We figure out this latency is introduced due to the query per second (QPS) limits of Kubelet to the API server (default is 5). After being aware of this, we find in additional tests that by increasing QPS limits, curve “running” gradually converges with “pod_running’, and results in much lower latency. Therefore the previous e2e test pod startup results reflect the combined latency of both Kubelet and time of uploading status, the performance of Kubelet is thus under-estimated.Figure 5. Time series page using data from build 162.Further, by comparing the time series data of build 162 (Figure 5) and build 173 (Figure 6), we find that the performance pod startup latency fluctuation actually happens during updating pod statuses. Build 162 has several straggler “pod_status_running” events with a long latency tails. It thus provides useful ideas for future optimization. Figure 6. Pod startup latency of build 173.In future we plan to use events in Kubernetes which has a fixed log format to collect tracing data more conveniently. Instead of extracting existing log entries, then you can insert your own tracing probes inside Kubelet and obtain the break-down latency of each segment. You can check the latency between any two probes across different builds in the “TRACING” page, as shown in Figure 7. For example, by selecting “pod_config_change” as the start probe, and “pod_status_running’ as the end probe, it gives the latency variance of Kubelet over continuous builds without status updating overhead. With this feature, developers are able to monitor the performance change of a specific part of code inside Kubelet. Figure 7. Plotting latency between any two probes.Future WorkThe node performance dashboard is a brand new feature. It is still alpha version under active development. We will keep optimizing the data collecting and visualization, providing more tests, metrics and tools to the developers and the cluster maintainers. Please join our community and help us build the future of Kubernetes! If you’re particularly interested in nodes or performance testing, participate by chatting with us in our Slack channel or join our meeting which meets every Tuesday at 10 AM PT on this SIG-Node Hangout.–Zhou Fang, Software Engineering Intern, GoogleDownload KubernetesGet involved with the Kubernetes project on GitHub Post questions (or answer questions) on Stack Overflow Connect with the community on SlackFollow us on Twitter @Kubernetesio for latest updates
Quelle: kubernetes

DockerCon Returns to Europe in 2017

DockerCon is making its return to Europe next year! DockerCon Europe will be held in the beautiful city of Copenhagen, Denmark at Bella Center Cope
nhagen from October 16 – October 18, 2017. We plan on opening the week on Monday, October 16 with paid trainings and workshops, then General Session will kick off the conference the morning of Tuesday October 17 and the conference will continue through Wednesday October 18.
Three reasons why we are excited about DockerCon Europe in Copenhagen
 

On behalf of the entire Docker team, it’s safe to say that we cannot wait to reunite with the Docker Community in Europe under one roof again! Local Docker Meetup chapters take place every week to fuel the community enthusiasm, but there is something special about coming together for DockerCon and collaborating, learning and networking as a big group.

Recently remodeled in 2014/2015, the Bella Center Copenhagen is an ultra-modern event space featuring Scandinavian design throughout including open space with lots of indoor greenery. Bella Center Copenhagen is also one of the most sustainable venues in the world. They practice waste sorting in 16 categories, have an 850 kW wind turbine on-site for energy, as well as a living roof that is home to one million bees!

Another fun fact: Did you know that over one million people visit the mermaid statue in Copenhagen, inspired by Hans Christian Andersen’s The Little Mermaid fairytale? With all of the aquatic references throughout the city, we can’t wait to see what Moby scenes Docker illustrator Laurel will come up with!

Stay tuned in the upcoming months for both DockerCon US (April 17-20, 2017 in Austin, Texas) and DockerCon EU news including ticket sales and call for papers. In the meantime, catch up on all past DockerCon action and be sure to save the dates!
 

The post DockerCon Returns to Europe in 2017 appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/