da Agency

Announcing Google Cloud Public Sector Summit, a free global digital event: Dec. 8-9

2020 has been a year of rapid change, unexpected challenges and new frontiers. In the face of this, government agencies, local leaders, and educational institutions worldwide have been challenged to reinvent their engagement with citizens, the way they accomplish their missions, and how they educate students. Despite its challenges, the pandemic has proven to be a crucible for rapid innovation and progress. Government and academic leaders have done heroes’ work, in some cases consolidating years’ worth of digital transformation into just days and weeks. They’ve spun up testing centers, unemployment websites, new citizen-facing call centers and digital classrooms, in some cases overnight, all while maintaining continuity of government services.At Google Cloud, we’re privileged to work side-by-side with our public sector customers during this time, as they respond to—and gradually begin to recover from—COVID-19. The pandemic has been a reminder of the incredible power of public and private sector partnership, and the positive impact we make together. With that spirit in mind, I’m thrilled to announce our first-ever Google Cloud Public Sector Summit. We invite all of our public sector customers across the globe to join us for a special, two-day virtual event, Dec. 8-9, where we’ll share lessons learned throughout this year and discuss the future of digital service. Here’s what you can expect at the Google Cloud Public Sector Summit:Hear from industry leaders: Join our keynotes and breakout sessions to hear industry experts discuss the latest trends and get a glimpse into the future of public sector work.Gather insights from your peers: Attend breakout sessions where government and education leaders will share their successes around fulfilling critical missions, scaling IT services, and building cultures of innovation. Engage with Google experts and partners: Request time for you or your team to engage live with Google solution experts and Google partners. Learn about Google’s purpose-built solutions for education and government: Hear about our security and compliance offerings and other solutions, including our student success platform, citizen services, workforce collaboration, cyber analytics, and more.The Google Cloud Public Sector Summit begins Dec. 8. Register today, at no cost, on the Public Sector Summit event website.
Quelle: Google Cloud Platform

da Agency

Understand production performance with Cloud Profiler history view

Cloud Profiler is a favorite of Google Cloud customers thanks to the insight that it provides into the performance of your production code. You can use this knowledge to reduce and shorten outages, improve performance, and optimize compute spend—always a popular topic! Profiler has always provided the ability to view and compare CPU and memory performance over time through time filters and the comparison feature. Now, Profiler lets you view the performance of a single function or a group of functions over time.You can access this using the history view button, which displays the new function history view window that shows the relative resource consumption of the most resource hungry functions within a given service.You can use the history view feature to do a number of different things:Intuitively understand the performance trends of the most resource-intensive functions over timeDiscover unintended resource usage changes. Large-scale or complex code changes can have unintended resource usage implications. Looking at the history view, you can verify that your code changes didn’t introduce unintended performance or resource usage changes.Verify the performance impact of recent code changes.Rapidly find the root cause of an outage of severe performance regression. If a service has stalled due to high CPU or memory consumption, looking at the history of each function’s resource consumption will provide insight into when and what parts of the code recently started using more resources.Characterize how a codebase’s performance changes due to external factors such as usage spikes (this can also be visualized with Profiler’s weight filter) or known changes in usage patterns.Profiler history view is available immediately as a preview for all users. Read more about how to use this feature.If you are new to Profiler, our Quickstart or codelab can help you get started.
Quelle: Google Cloud Platform

da Agency

A deeper dive into Confidential GKE Nodes—now available in preview

The benefits of containers and Kubernetes over traditional on-premises architectures are well-documented and understood. But when considering moving to the cloud, organizations want controls to limit risk and potential exposure of their data.In July, we announced the availability of the Confidential Computing product family, whose  breakthrough technology encrypts data in-use—while it is being processed—without any code changes to the application. We also introduced Confidential VMs as the first member of that product family, which perform at levels comparable to VMsA few weeks back we announced the upcoming launch of Confidential Google Kubernetes Engine (GKE) Nodes in preview. Today, as we kick off cybersecurity month, we are rolling out the preview for Confidential GKE Nodes. With Confidential GKE Nodes you can achieve encryption in-use for data processed inside your GKE cluster, without significant performance degradation. Built on Confidential VMs, which utilize the AMD Secure Encrypted Virtualization (SEV) feature, Confidential GKE Nodes encrypt the memory of your nodes and the workloads that run on top of them with a dedicated per-Node instance key that is generated and managed by the AMD Secure Processors, which is embedded in the AMD EPYC™ processor. These keys are generated by the AMD Secure Processor during node creation and reside solely within it, making them unavailable to Google or any VMs running on the host. This, combined with other existing solutions for encryption at rest and in-transit, and workload isolation models such asGKE Sandbox, provides an even deeper and multi-layer defense-in-depth protection against data exfiltration attacks. Confidential GKE Nodes also leverage Shielded GKE nodes to offer protection against rootkit and bootkits, helping to ensure the integrity of the operating system you run on your Confidential GKE Nodes.Enabling Confidential GKE NodesWhen creating a new cluster, you can enable Confidential GKE Nodes by specifying the –enable-confidential-nodes option:After you create a Confidential GKE cluster, all the nodes and node pools you create will be confidential.You can verify that your cluster is using Confidential GKE Nodes by using the describe command:gcloud beta container clusters describe [CLUSTER_NAME]If Confidential GKE Nodes are enabled, the output of the command will include these lines:confidentialNodes:enabled: trueEnabling applications to run with Confidential GKE NodesYou may be wondering what you need to change in your application to leverage Confidential GKE Nodes? The answer is nothing! Google’s approach to confidential computing is to enable an effortless lift and shift for existing applications so that all GKE workloads you run today can run on Confidential GKE Nodes without any code changes. Optionally, if you use a GitOps model for storing your application configurations, you can use the cloud.google.com/gke-confidential-nodes nodeSelector to declaratively  ensure that your sensitive workloads can only be scheduled on Confidential GKE Nodes. This can be useful later on if you want to demonstrate to auditors that your workloads ran exclusively on Confidential GKE nodes:Tune in to our latest Google Cloud Security Talks to learn more about confidential computing, and other areas of cloud security.Related ArticleExpanding Google Cloud’s Confidential Computing portfolioGoogle Cloud Confidential Computing is now GA and including Confidential GKE Nodes.Read Article
Quelle: Google Cloud Platform

da Agency

Turn any Dataflow pipeline into a reusable template

As data analysis grows within an organization, business teams need the ability to run batch and streaming jobs and leverage the code written by engineers. But re-running existing code often requires setting up a development environment and making slight code changes, which is challenging for people without a programming background.With this challenge in mind, we recently introduced Dataflow Flex Templates, which make it even easier to turn any Dataflow pipeline into a reusable template that anyone can run. Existing classic templates let developers share batch and streaming Dataflow pipelines via templates so everyone can run a pipeline without a development environment or writing code. However, classic templates were rigid for a couple of reasons:First, since the Dataflow pipeline execution graph is permanently fixed when the developer converts the pipeline into a shareable template, classic templates could then only be run to accomplish the exact task the developer originally had in mind. For example, choosing a source to read from, such as Cloud Storage or BigQuery, had to be determined at the template creation stage and could not be dynamic based on a user’s choice during template execution. So developers sometimes had to create several templates with minor variations (such as whether the source was Cloud Storage or BigQuery). Second, the developer had to select the pipeline source and sink from a limited list of options because of classic templates’ dependency on the ValueProvider interface. Implementing ValueProvider allows a developer to defer the reading of a variable to whenever the template is actually run. For example, a developer may know that the pipeline will read from Pub/Sub but wants to defer the name of the subscription for the user to pick at runtime. In practice, this means that developers of external storage and messaging connectors needed to implement Apache Beam’s ValueProvider interface to be used with Dataflow’s classic templates.The new architecture of Flex Templates effectively removes both limitations, so we recommend using Flex Templates moving forward.Flex Templates bring more flexibility over classic templates by allowing minor variations of Dataflow jobs to be launched from a single template and allowing the use of any source or sink I/O. Since the execution graph is now built dynamically when the template is executed (instead of during the template creation process), minor variations can be made to accomplish different tasks with the same underlying template, such as changing the source or sink file formats. Flex Templates also remove the ValueProvider dependency, so any input and output source can be used.Next, we’ll offer a developer’s guide to why and how to create custom Flex Templates.Why sharing Dataflow pipelines has been challengingAn Apache Beam pipeline commonly reads input data (from the source), transforms it (using transforms like ParDo) and writes the output (to the sink):A simple Dataflow pipelinePipelines can be significantly more sophisticated, with multiple input sources, series of chained transforms (DAG of steps), and multiple output sinks. Once an Apache Beam pipeline is constructed, it can be deployed and executed in various runners such as Dataflow, Spark, Flink or Direct Runner (for local runs). Templates are a Dataflow-specific feature that makes it easier to re-run pipelines on the Dataflow runner.But what exactly does “running a pipeline” mean? When you run a Dataflow pipeline, the Apache Beam SDK executes the code locally and builds an execution graph converting the sources, sinks and transforms into nodes. The execution graph object is then translated (serialized) into JSON format and submitted to the Dataflow service. Finally, the Dataflow service performs several validations (API, quota and IAM checks), optimizes the graph, and creates a Dataflow job to execute the pipeline.Sharing a Java-based Dataflow pipeline before TemplatesPrior to Dataflow templates, it was challenging for developers to share pipelines with coworkers. In the past, a developer would start by creating a development environment (with the JDK, Apache Beam SDK, and Maven or Gradle typically installed for Java or Python, and pip typically installed for Python), write the code, build a binary artifact (fat JAR with dependencies or Python equivalent) and share the artifact either through an artifactory or Cloud Storage. Users would then set up local runtime environments and fetch the binary into their individual environments for execution. If the pipeline was written in Java, the runtime environment would need Java JRE installed; if the pipeline was Python-based, all Python packages the developer used would need to be installed. Finally, when the user ran the pipeline, an execution graph would get generated and sent to the Dataflow service to run the pipeline on cloud. There were several points where something could break in these steps, and creating a runtime environment was a non-trivial task for users without a technical background. Even scheduling pipelines on a VM (using cron) or third-party schedulers needed a similar runtime environment to exist, complicating the automation process.Sharing Dataflow pipelines with classic templatesSharing a Dataflow pipeline with classic templatesClassic templates significantly improve the user experience for rerunning Dataflow pipelines. When the developer runs the pipeline code in the development environment, the pipeline now gets converted into a Dataflow template stored on Cloud Storage. The staged template consists of the translated JSON execution graph along with dependencies (pipeline artifacts and third-party libraries). The execution graph is permanently fixed and the user cannot change the shape of the DAG. Once the Cloud Storage bucket permissions have been adjusted to share with users, they can invoke the pipeline while passing in any required parameters directly via a gcloud command, a REST API, or the Dataflow UI in Google Cloud Console. Users no longer need to build and configure a runtime environment. Cloud Scheduler can also be used to easily trigger the pipeline to be run on a regular schedule without the need of a runtime environment.Sharing Dataflow pipelines with Flex TemplatesSharing a Dataflow pipeline with Flex TemplatesSimilar to classic templates, with Flex Templates staging and execution are still separate steps. However, the runnable pipeline artifact that gets staged is different; instead of staging a template file in Cloud Storage, developers now stage a Docker image in Google Container Registry.Additionally, a developer does not need to run the pipeline to create a Flex Template. Instead, the developer packages the pipeline code/binaries, including dependencies, into a Docker image and stores it in Container Registry, then creates a template spec file stored in Cloud Storage.Four steps in the developer workflowThe staged image is built using a Google-provided base image and contains the pipeline artifacts with dependencies and environmental variables:Components inside the Flex Template Docker imageThe Docker image does not contain the JSON serialized execution graph. For Java-based pipelines, the image contains the JAR file; for Python pipelines, the image contains the Python code itself. Only when a user actually runs the Flex Template does the graph construction phase start within a new container and the execution graph is constructed based on the parameters the user provides at runtime. This allows for execution graphs to be dynamically constructed based on final input parameters from the user.The file in Cloud Storage is not the Flex Template, but rather the template spec file. This spec file contains all of the necessary information to run the job, such as the Container Registry image location, SDK language, metadata such as the name and description of the template and any required or optional parameters the template needs. Similar to classic templates, regex can be used to validate the input parameters provided by the user.Users can execute the Flex Template using a gcloud command, calling the REST API, or using the Dataflow UI in Google Cloud Console referring to a template spec file stored in Cloud Storage and providing required parameters. Automating and scheduling a recurring job can also be done via Cloud Scheduler or Terraform (support for Airflow is under development).Comparing classic vs. Flex TemplatesThe following table summarizes the similarities and differences between classic and Flex templates:Create your first Dataflow Flex TemplateIf you are new to Dataflow templates, we recommend starting with our Google Cloud-provided templates for moving data between systems with minimal processing. These are production-quality templates that can be easily run from the Dataflow UI in Google Cloud Console.If you want to automate a task that is not in the provided templates, follow our Using Flex Templates guide. The tutorial walks you through a streaming pipeline example that reads JSON-encoded messages from Pub/Sub, transforms message data with Beam SQL, and writes the results to a BigQuery table.You can also review the source code for the Google-provided templates and review our examples for generating random data, decompressing data in Cloud Storage, analyzing tweets, or doing data enrichment tasks like obfuscating data before writing it to BigQuery.Finally, when you’re ready to share the Flex Template with users, the Google Cloud Console UI provides an option to select a Custom Template and then asks for a Cloud Storage path of its location:Creating a custom template from Google Cloud ConsoleLearn more about Dataflow on our site, and check out our presentation on Flex Templates at Beam Summit.Thanks to contributors to the design and development of the new release of Dataflow Flex Templates, in no particular order: Mehran Nazir, Sameer Abhyankar, Yunqing Zhou, Arvind Ram Anantharam, Runpeng Chen, and the rest of the Dataflow team.
Quelle: Google Cloud Platform

da Agency

Why build apps on a cloud-native database like MongoDB Atlas?

IT departments today are being challenged to adopt new and far more strategic roles within the organizations they serve. As more businesses turn to technology as a competitive differentiator, IT is being challenged to step away from its traditional focus on running infrastructure and to step up as a strategic business function—delivering software and services that support innovation and create compelling customer experiences.Modern, cloud-native applications are critical to this transformation. The cloud allows IT organizations to retire legacy applications and infrastructure, and to deliver software that is far more resilient and scalable—without sacrificing usability and performance.The legacy database: a weak link in the cloud-native chainAs businesses migrate to managed infrastructure and cloud-native apps, on-prem legacy database systems have emerged as a serious barrier to scalability and performance. As a result, more businesses are turning to new database options such as MongoDB Atlas that are designed to meet the demands of modern, cloud-native environments.Here’s the problem in a nutshell: Legacy databases were designed for environments where data came in small, tidy packages and where scalability wasn’t a major requirement. That makes these systems a poor fit for cloud-native applications that are built to scale and that drive massive amounts of data.JSON-based document databases like MongoDB, on the other hand, are very well-suited for modern application development methods: teams can store data in a format not unlike the objects in their code, allowing them to work quickly and efficiently. And as a managed database-as-a-service solution, MongoDB Atlas gives IT organizations an alternative to the cost and complexity of an on-premises, legacy system.AutoTrader UK builds a future on MongoDB AtlasAutoTrader UK is a great example of an IT organization that’s using MongoDB Atlas on Google Cloud to modernize its legacy databases in favor of a fully managed, database-as-a-service platform. AutoTrader UK relied heavily on Oracle and SQL Server but began using MongoDB in its own data centers so that its development teams could move faster. After migrating to MongoDB Atlas, the company retained all of the advantages of its self-managed MongoDB environment, but shed the operational and infrastructure complexity. This was a critical move for a company built almost entirely on the value of its business data, and where scalability and resource management posed especially pressing challenges.To release new features faster, AutoTrader UK launched multiple initiatives to improve team efficiency and agility, including migrating entirely to the cloud and moving off legacy databases. Their team already had experience with and enjoyed using various Google Cloud services like Dataflow and BigQuery, which helped fuel their decision to migrate entirely to Google Cloud – replacing their Oracle database with Cloud SQL – and MongoDB Atlas. As Russell Warman, Head of Infrastructure at AutoTrader put it: “From a business perspective, migrating to Google Cloud Platform means we can get ideas up and running quickly, enabling us to build brilliant new products, helping us to continue to lead in this space.”The company’s developers now roll out new products far more quickly and with greater confidence, and the company has since made big strides in decommissioning its on-premises data center. This brought IT’s legacy management burden under control, along with infrastructure and related costs.Just as important, the move to MongoDB Atlas and Cloud SQL set the stage for AutoTrader UK to compete and win with software. The company’s development team pushed over 36,000 releases live in a year including more than 450 releases in a single day. With nearly 270 apps deployed in the public cloud today, AutoTrader UK maintains a 99.79% release success rate and 99.99% availability for its core search functionality.The benefits of MongoDB Atlas MongoDB Atlas brings together capabilities that are critical to a modern, cloud-native, microservice-aligned database architecture, including:Developer productivity: MongoDB Atlas is a non-relational database that employs a JSON-based document data model. MongoDB documents map naturally to an object-oriented programming model, which makes it intuitive and easy to work with using any object-oriented language. This flattens the learning curve when a dev team builds applications with MongoDB Atlas. Many developers find MongoDB especially flexible as fields can vary from document to document and data structure can be easily changed over time.Scalability: MongoDB Atlas allows IT to deploy right-sized applications as a matter of course; it scales up or down instantly and on demand, without risking application downtime. By relying on sharding, MongoDB Atlas avoids issues with hardware bottlenecks while also minimizing the complexity that often crops up at scale. MongoDB Atlas users can also select from a number of sharding strategies based on the workloads and query patterns they need to serve.Availability and uptime: Running an application in the public cloud usually offers better availability from the get-go compared to an on-premises environment. This is because of the massive investments providers such as Google Cloud make to replicate their infrastructure across multiple geographical regions, and to gain other capabilities that very few businesses have the resources to duplicate.Building on these capabilities, MongoDB Atlas deploys every database cluster as a self-healing replica set that fails over automatically when necessary. MongoDB Atlas will automatically provision replica set members across multiple availability zones within a region—a critical safeguard against the most common, localized failures that create the greatest risk for most businesses. And when a MongoDB Atlas instance fails, the system recovers instantly and automatically in most cases.Automation: “Keeping the lights on” is a massive source of waste and frustration when teams deal with legacy database systems. MongoDB Atlas automates key tasks during provisioning and configuration, maintenance, and disaster recovery processes so teams don’t waste time on mundane maintenance and upkeep. It also employs automated monitoring and alerting to help teams detect and troubleshoot performance issues before they affect your applications or user experience.A modern database designed to adaptBy integrating within a modern, cloud-native app environment, MongoDB Atlas gives developers the freedom to architect and re-architect applications as an organization’s business needs change—without the risk of outgrowing a legacy database solution or of enduring a forced product upgrade to accommodate growth. Leaving behind a legacy database and moving to MongoDB Atlas can be a big step towards achieving this goal.Learn more about MongoDB Atlas on Google Cloud.Watch Vadim Supitskiy, Forbes’ CTO chat with Lena Smart, MongoDB CISO about how Forbes set digital innovation standards with MongoDB and Google Cloud.Related ArticleHow our customers are using MongoDB Atlas on Google Cloud PlatformOpen-source cloud databases can run workloads quickly. MongoDB Atlas on GCP brings you combined billing, management and ease of use.Read Article
Quelle: Google Cloud Platform