März 2020 - Seite 85 von 89 - Cloud Computing Köln

I’ve been committed to open source software for over a decade because it fosters a deep collaboration across the developer community, resulting in ground-breaking innovation. At the heart of open source is the freedom to learn from each other and share ideas, empowering the brightest minds to work together on the cutting edge of software development.

Over the last decade, Microsoft has become one of the largest open source contributors in the world, adding to Hadoop, Linux, Kubernetes, Python, and more. Not only did we release our own technologies like Visual Studio Code as open source, we have also collaborated and contributed to existing open source projects. One of our proudest moments was when we became the release masters for YARN in late 2018, having open sourced over 150,000 lines of code, which enabled YARN to run on clusters 10x larger than before. We're actively growing our community of open source committers within Microsoft.

We’re constantly exploring new ways to better serve our customers in their open source journey. Our commitment is to combine the innovation open source has to offer with the global reach and scale of Azure. Today, we're excited to share a few important updates to accelerate our customers’ open source innovation.

Microsoft supported distribution of Apache Hadoop

Microsoft has been an early supporter of the Hadoop ecosystem since the launch of HDInsight in 2013. With HDInsight, we have been focused on delivering seamless integration of key Azure services like Azure Data Factory and Azure Data Lake Storage, with the power of the most popular open source frameworks to enable comprehensive analytics pipelines. To accelerate this momentum, we're pleased to share a Microsoft supported distribution of Apache Hadoop and Spark for our new and existing HDInsight customers. This distribution of Apache Hadoop is 100 percent open source and compatible with the latest version of Hadoop. Users can now provision a new HDInsight cluster based on Apache code that is built and wholly supported by Microsoft.

By providing a Microsoft supported distribution of Apache Hadoop and Spark, our customers will benefit from enterprise-grade security features like encryption, and native integration with key Azure stores and services like Azure Synapse Analytics and Azure Cosmos DB. Best of all, given that Microsoft directly supports this distribution, we can quickly provide support and upgrades to our customers and deliver the latest innovation from the Hadoop ecosystem. All of this will enable customers to innovate faster, without being restricted to proprietary technology just to use our support and features. Additionally, Azure will continue to develop a vibrant marketplace of open source vendors

“We at Cloudera welcome the commitment from Microsoft to Apache Hadoop and Spark. Open-source is key to our mutual customers’ success. Microsoft’s initiative represents a strong endorsement of open-source for the enterprise and we are excited to continue our partnership with Cloudera Data Platform for Microsoft Azure.” Mick Hollison, Chief Marketing Officer at Cloudera

This is part of our strong commitment to Hadoop, open source analytics, and the HDInsight service. In addition to our deeper engagement in supporting open source Hadoop and Spark, in the coming months, we’ll enable the most requested features on HDInsight that lower costs and accelerate time to value. These include an improved provisioning and management experience, reserved instance pricing, low-priority virtual machines, and auto-scale.

We have always sought to meet customers where they are, from our decision four years ago to support HDInsight solely on Linux, to our recent migration of clusters distribution in-house. Customers don't need to take any specific actions to benefit from these changes. These upcoming improvements to HDInsight will be seamless and automatic, with no business interruption or pricing changes.

Welcome new PostgreSQL committers

Since the Citus Data acquisition, we have doubled down on our PostgreSQL investment based on the tremendous customer demand and developer enthusiasm for one of the most versatile databases in the world. Today, Azure Database for PostgreSQL Hyperscale is generally available, and it’s one of our first Azure Arc-enabled services.

The innovation and ingenuity of PostgreSQL continue to inspire us, and it would not be possible without the contribution and passion of a dedicated community. We will continue to contribute to PostgreSQL. Recently, we contributed pg_autofailover to the community to share our learnings of operating PostgreSQL at cloud scale.

To build on our investment in PostgreSQL, we're excited to welcome Andres Freund, Thomas Munro, and Jeff Davis to the team. Together, they bring a decade of collective experience and a leading track record as core committers to PostgreSQL. They, like the rest of the team, are engaging with and listening to the global Postgres community, as we work to deliver the best of cloud scale, security, and manageability to open source innovation.

We're committed to actively engaging the open source community and providing our customers with choice and flexibility. The true open source spirit is about collaboration, and we’re excited to combine the best of open source software with the breadth of Azure. Most importantly, we are bringing together the best minds and talented visionaries, both at Microsoft and in the broader open source community, to constantly improve our open source products and deliver the newest features to our customers. Here’s to open source!

Additional resources

HDInsight Documentation is your one-stop-shop for learning all about this analytics platform.
PostgreSQL Committers Blog: Visit to learn more about the three new committers we hired.

Quelle: Azure

3. März 2020

da Agency

How to detect and prevent network outages—and stay compliant too

By some estimates, 75% of network outages and performance issues are the result of a misconfiguration, and more often than not, these misconfigurations aren’t discovered until they’re in production. That’s stressful for network administrators and architects—not knowing the impact of a configuration change in firewall rules or routing rules makes network monitoring reactive rather than proactive, introduces risk and leads to long troubleshooting times. We recently introduced Network Intelligence Center, Google Cloud’s comprehensive network monitoring, verification and optimization platform that works across the cloud and on-premises data centers, including an initial set of modules that can predict and heal network failures. In this post, we’ll take a deep dive into the Connectivity Test module, which helps diagnose connectivity issues and predicts the impact of configuration changes, so you can better prevent outages. Connectivity Test enables you to self-diagnose connectivity issues within Google Cloud, or Google Cloud to an external end-point that is on-prem or even in another cloud. You can also create, save and run tests. With these capabilities, Connectivity Test can help you perform a variety of important network administration tasks such as: Understand and verify network design and architectureTroubleshoot and fix connectivity issuesVerify the impact of configuration changesEnsure network securityMake your security and compliance audits easier and more manageableWe’ll discuss each of these use cases in greater depth below, but first, let’s look at the Connectivity Test architecture. Connectivity Test technical overviewThe Connectivity Test module is powered by a network reachability analysis platform, which determines whether there’s connectivity between source and destination. If there’s no connectivity, Connectivity Test pin-points where it’s broken and identifies the root-cause, for example, a firewall rule blocking the connectivity. Rather than the traditional approach of looking at live traffic flows or sending traffic through the data plane, this reachability analysis platform uses a network verification approach based on formal verification techniques. It creates an accurate and comprehensive model of the network based on the current network design, configurations and network state. The model can reason about all possible behaviors and help troubleshoot configuration issues or prove compliance with an intended policy. Thus, network verification can exhaustively prove or disprove reachability in ways that traditional approaches cannot.Connectivity Test uses two key components in particular to perform this analysis. Data plane modelTo perform static reachability analysis, Connectivity Test relies on an idealized data plane model. In other words, Connectivity Test derives instances, networks, firewall rules, routes, VPN tunnels, etc. from GCP project configurations, which it then analyzes to verify whether two points can be reached. The most important configurations that it uses are VPC network properties, network services (load balancers), hybrid cloud configurations (VPN, Interconnect, Cloud Routers), and VM and Google Kubernetes Engine endpoint configurations. Network Abstract State MachineConnectivity Test also relies on a Network Abstract State Machine, an idealized model of how a Google Cloud VPC network processes packets. Specifically, Google Cloud processes a packet in several logical steps that are modeled as a finite state machine, which takes a bounded number of steps between discrete states until the packet has been delivered or dropped.The diagram below shows a model for how Connectivity Test simulates trace traffic between two VMs. Depending on your GCP network and resource configurations, this traffic could go through, for example, a Cloud VPN tunnel, a GCP load balancer, or a peered VPC network before reaching the destination VM.Simulating traffic between two VMs based on a network modelConnectivity Test in actionAs mentioned above, early Network Intelligence Center customers have been using Connectivity Test for five key use-cases. Let’s take a deeper look at each one.1. Understand and verify network design and architecture: As you migrate your workloads from on-prem to cloud, you want greater visibility into the network paths. You might want to check if traffic is going through a VPN or Interconnect, or which firewall or routing rules are getting applied between a source and destination endpoint. With Connectivity Test, you can see a complete trace of the packet flow from source to destination including all the hops (routes, egress/ingress firewall rules, VPN/Interconnect, VPC peering, NAT, and more), helping you better understand and easily verify the network design. You can even see multiple traces between source and destination, which is helpful for network configurations such as with High-Availability VPN.2. Troubleshoot and fix connectivity issues: Most network outages are the result of a misconfiguration, such as a badly designed firewall rule or an incorrect routing policy. In a complex cloud environment with shared VPCs and many firewall and routing configurations, it could take hours or days to troubleshoot connectivity issues and find the root cause. We see this frequently with customer support cases. With Connectivity Test, you can run a diagnostic test between the source and destination endpoint that is having a connectivity problem and quickly pinpoint the root cause, including the specific firewall rule or routing issue. You can then update the firewall rule and re-run the test(s) to verify if the firewall configuration update fixes the connectivity problem. This cuts down troubleshooting time from days or hours to minutes—and lets you very quickly root-cause, implement and then verify the fix.3. Verify the impact of configuration changes: In production environments, it’s imperative to to understand the impact of any configuration change before you deploy it, so you can catch any mistakes up front. With Connectivity Test, you can create a set of tests that reflect your connectivity intent, for example, that there should or should not be connectivity between a source and destination endpoint. After you make a configuration change, you can re-run a single test or all these tests to verify whether the connectivity intent hasn’t been violated. If you find that the intent has been violated, you can roll back the configuration change and re-run the test(s) to make sure your Connectivity stays as desired and expressed by your Intent in the test(s).4. Ensure network security: Connectivity Test helps ensure that your network configuration reflects your network security intent. For example, you may not want your web tier VMs to be connected to your database tier VMs, or database tier VMs should not be connected to the internet. You can express these security intents as Connectivity Tests, to verify that the endpoints are indeed “unreachable,” meaning the desired isolation exists. You can then run these tests periodically to detect any security violations.5. Make your security and compliance audits easier and more manageable: In the above example, we discussed how you could express your security intent through Connectivity Tests and detect and fix any violations that are caught. If your security and compliance audits have any specific rules, policies and/or intents that you need to comply with, you can create a set of tests, run the tests periodically and log your results. Then, during a security and compliance audit, you have a ready-to-go log of how your network performed against the audit’s requirements.All the above use-cases can be realized through an easy-to-use UI. You can also run tests from the command line or an API, to run the tests automatically as part of your CI/CD pipeline.What customers are sayingEarly Network Intelligence Center adopters report that the Connectivity Test module helps them find and resolve problems faster. Randstad,an employment agency, says that Connectivity Test has become a key part of its networking toolset. “We are excited about using Network Intelligence Center for troubleshooting network connectivity issues. We have become frequent users of Connectivity Test within Network Intelligence Center to resolve connectivity issues both within GCP configuration, and to confirm when the issue is outside GCP. It has reduced total troubleshooting effort, and saves us significant time.” – Kevin Scott, Senior Director, IT Infrastructure & Architecture, RandstadMeanwhile, managed service provider HIPAA Vault uses Connectivity Test to check and demonstrate compliance. “We’re excited that with the help of Network Intelligence Center, we’ll be able to verify that our network connectivity matches intent and quickly troubleshoot network configuration issues.” – David Breise, Cloud and Network Engineer, HIPAAVault. To learn more about how you can leverage Network Intelligence Center and Connectivity Test to improve the performance and availability of your network, watch this video.
Quelle: Google Cloud Platform

3. März 2020

da Agency

With Kubeflow 1.0, run ML workflows on Anthos across environments

Google started the open-source Kubeflow Project with the goal of making Kubernetes the best way to run machine learning (ML) workloads in production. Today, Kubeflow 1.0 was released. Kubeflow helps companies standardize on a common infrastructure across software development and machine learning, leveraging open-source data science and cloud-native ecosystems for every step of the machine learning lifecycle. With the support of a robust contributor community, Kubeflow provides a Kubernetes-native platform for developing, orchestrating, deploying, and running scalable and portable ML workloads.Using Kubeflow on Google Cloud’s Anthos platform lets teams run these machine-learning workflows in hybrid and multi-cloud environments while taking advantage of Google Kubernetes Engine’s (GKE) enterprise-grade security, autoscaling, logging, and identity features. Barton Rhodes, Senior Machine Learning Engineer at DaVita, and an early user of Kubeflow on Anthos, said the enterprise features introduced in Kubeflow 1.0 will make a big difference for his organization: Having used Kubeflow (since 0.1) as a development foundation for a platform of several teams of data scientists needing to operate in hybrid-cloud environments, it has been a pleasure and an inspiration to see the project mature. When so much of the ethics and impacts of machine learning come down to the details of implementation, operations, safety, and reproducibility for the resulting artifacts, open source allows the broader community to build and tackle these challenges on top of shared foundations. With this release and exciting new features like multi-user isolation, workload identity, and KFServing, it is that much easier to introduce Kubeflow or its individual resources into the enterprise. The blog post introducing Kubeflow 1.0provides a technical deep-dive into the core set of applications included in the open-source release. In this post, we’ll look at more details on the advantages of using Kubeflow 1.0 on Anthos for the enterprise. SecurityFor data scientists to be productive, they need easy and secure access to UIs like the Kubeflow dashboard, Jupyter UI, and TensorBoard.When you deploy Kubeflow on Anthos, it can be secured using Identity-Aware Proxy (IAP), Google Cloud’s zero trust access solution (also known as BeyondCorp). Using IAP, you can restrict access to Kubeflow based on either IP (e.g. to your corporate network), device attributes (e.g. to ensure Kubeflow is only accessed from up-to-date devices), or both.AutoscalingWhen deployed on Anthos, Kubeflow takes advantage of GKE autoscaling and node auto-provisioning to right-size your clusters based on your workloads. If the existing node pools have insufficient resources to schedule pending workloads, node auto-provisioning will automatically create new ones. For example, node auto-provisioning will automatically add a GPU node pool when a user requests a GPU. Autoscaling can also add more VMs to existing node pools if there’s insufficient capacity to schedule pending pods.LoggingGKE has direct integration with Cloud Logging, ensuring that the logs from all of your workloads are preserved and easily searchable. As this MNIST example shows, by using a query like the one below, you can fetch the logs for one of the pods in a distributed TensorFlow job by filtering based on the pod label.Cloud Logging’s integration with BigQuery makes it easier to begin collecting the metrics you need to evaluate performance. If your application emits logs as JSON entries, they will be indexed and searchable in Python. You can then leverage Cloud Logging’s export functionality to export them to Cloud Storage or BigQuery to facilitate analysis.Combining BigQuery logging with Kubeflow notebooks can help you analyze model performance. This GitHub notebook illustrates how the Kubeflow project is using this combination to measure the performance of models that automatically classify Kubeflow issues. Using pandas-gbq we can more easily generate Pandas Dataframes based on SQL queries, then analyze and plot results in our notebooks. Below is a snippet illustrating how you can log predictions from python.Here we’re using Python’s standard logging module with a custom formatter to emit the logs as serialized JSON. The structure is preserved when the logs are ingested into Cloud Logging and then exported to BigQuery, and we can search based on the extra fields that are provided.Workload IdentityOn Anthos, Kubeflow uses Workload Identity to help seamlessly integrate your AI workloads running on GKE with Google Cloud services. When you create a Kubeflow namespace using Kubeflow’s profile controller, you can select a Google Cloud service account to bind to Kubernetes service accounts in the resulting namespace. You can then run pods using those Kubernetes service accounts to access Google Cloud services like Cloud Storage and BigQuery without requiring additional credentials.The MNIST example mentioned above relies on workload identity to let your Jupyter notebooks, TFJobs, and Kaniko Jobs talk to Cloud Storage. What’s nextKubeflow 1.0 is just the beginning. We’re working on additional features that will help you be more secure and efficient. Here’s what you can look forward to in upcoming releases:Support for running ML workloads on-prem using AnthosUsing Katib and Batch on GKE to run large-scale hyperparameter tuning jobsA solution for preventing data exfiltration by deploying Kubeflow with private GKE and VPC Service ControlsGet startedTo get started with Kubeflow on Anthos, check out this tutorial. It walks through every step you need to deploy Kubeflow on Anthos GKE and then run MNIST E2E.
Quelle: Google Cloud Platform

3. März 2020

da Agency

Off to a fast start in 2020 with over 70 new Partner Specializations

As we embark on a new decade of the ‘20s, there’s no question that our greatest opportunities are ahead of us. The passion we see from our partners fuels our drive to help customers succeed, and we remain committed to supporting our customers and partners in theirjourney to differentiation. It’s amazing to see our partners growing their businesses with us, and even more incredible to witness our partners driving real business value for our customers with Google Cloud.We are aligning our Google Cloud solutions with the needs of our customers and giving our partners the ability to stand out with focus and precision. This will make it easier for customers to learn more about our partners and clearly understand partner capabilities. For example, achieving a Google Cloud Specialization is not an easy task. Partners understand the value of investing in such a rigorous process and how beneficial their Specialization badge is because customers are reassured that they’ve been vetted by Google.We’re very excited to announce that partners achieved more than 70 new Specializations in Q4 2019. Congratulations to everyone involved, we know that the ability to differentiate your business through Expertise and Specialization is one of the many reasons you have chosen to grow with Google Cloud.As we look ahead at our mutual goals in 2020 and beyond, the key to growth will be the distinction of our business solutions, our commitment to expertise in deployment, and best practices for customer success. Together, we will solve today’s business challenges, while better serving our customers and partners at greater speed and agility. Commendation to our partners and their supporting teams who have worked so hard and have achieved such a momentous milestone last quarter in the following practices:Application DevelopmentCI&T | Davinci Technologies | GoPomelo | Informatica El Corte Ingles | IPNET | Noovle | SADA | Shortcut AS | Solstice | Techolution | Tempus Nova, LLC.Cloud MigrationCognizant | Epam Systems, Inc. | Nortal | Onix | RedaptData AnalyticsAgile GCP Labs | Core Compete | Davinci Technologies | Epam Systems, Inc. | Grupodot | Leega Consultoria | Nubosoft | Pandera Systems LLC | Servinformacion | SoftServe | SpringML | Zenta GroupEducationForward EdgeInfrastructureAmarello | Arctiq Inc. | CI&T | Cloudypedia | Grid Dynamics International, Inc. | Incentro | MediaAgility | Pandera Systems LLC | Taos | ZazmicInternet of Things (IoT)SOTECLocation-Based ServicesApplied Geographics, Inc. (AppGeo) | Globema | ProgisMachine LearningComputas | Kasna | MediaAgility | Pluto7 Consulting Inc | SantoDigital | Servian | TWT Business SolutionsMarketing AnalyticsConverteo | Crystalloids | SpringML | StackProsSecurityGFT | Maven Wave | RackspaceTrainingAgilitics Pte. Ltd. | Fast Lane Institute for Knowledge Transfer / ITLSWork TransformationHiView Solutions | Master Concept | Nubosoft | Nuva S.A.S. | Qi Network | Revevol Group | Shivaami | Suitebriar, Inc.Work Transformation EnterpriseAgosto | Cloudbakers | Devoteam | Noovle | WurstaLooking for a partner in your region who has achieved an expertise and/or specialization? Search our global Partner Directory. For our partners who are ready to go to the next level, visit Google Cloud Expertise and Specialization to learn more.For a list of Google Cloud partners with Specializations view here.Not yet a Google Cloud partner? Visit Partner Advantage and learn how to become one today!
Quelle: Google Cloud Platform