Introducing Pay-as-you-go pricing for Apigee API Management

Apigee is Google Cloud’s API management platform that enables organizations to build, operate, manage and monetize their APIs. Customers from industries around the world trust Apigee to build and scale their API programs.While some organizations operate with mature API-first strategies, others might still be working on a modernization strategy. Even within an organization, different teams often end up with diverse use cases and choices for API management. From our conversations with customers, we are increasingly hearing the need to align our capabilities and pricing with such varied workloads.We’re excited to introduce a Pay-as-you-go pricing model to enable customers to unlock Apigee’s API management capabilities whilst retaining the flexibility to manage their own costs. Starting today, customers will have the option to use Apigee by paying only for what they are using. This new pricing model is offered as a complement to the existing Subscription plans (or) the ability to evaluate it for free.Start small, but powerful with Pay-as-you-go pricingThe new Pay-as-you-go pricing model offers flexibility for organizations to: Unlock the value of Apigee with no upfront commitment: Get up and running quickly without any upfront purchasing or commitmentMaintain flexibility and control in costs: Adapt to ever-changing needs whilst maintaining low costs. You can continue to automatically scale with Pay-as-you-go or switch to Subscription tiers based on your usageProvide freedom to experiment: Every API management use case is different and with Pay-as-you-go you can experiment with new use cases by unlocking value provided by Apigee without a long term commitmentPay-as-you-go pricing works just like the rest of your Google Cloud bills, allowing you to get started without any license commitment or upfront purchasing. As part of the Pay-as-you-go pricing model, you will only be charged based on your consumption of Apigee gateway nodes: You will be charged on your API traffic based on the number of Apigee gateway nodes (a unit of environment that processes API traffic) used per minute. Any nodes that you provision would be charged every minute and billed for a minimum of one minute.API analytics: You will be charged for the total number of API requests analyzed per month. API requests, whether they are successful or not, are processed by Apigee analytics. Analytics data is preserved for three months.Networking usage: You will be charged on the networking (such as IP address, network egress, forwarding rules etc.,) based on usageWhen is Pay-as-you-go pricing right for me?Apigee offers three different pricing modelsEvaluation plan to access Apigee’s capabilities at no cost for 60 days Subscription plans across Standard, Enterprise or Enterprise plus based on your predictable but high volume API needsPay-as-you-go without any startup costsSubscription plans are ideal for use cases with predictable workloads for a given time period, whereas Pay-as-you-go pricing is ideal if you are starting small with a high value workload. Here are a few use cases where organizations would choose Pay-as-you-go if they want to:Establish usage patterns before choosing a Subscription modelEvolve their API program by starting with high value and low volume API use casesManage and protect your applications build on Google cloud infrastructureMigrate or modernize your services gradually without disruptionNext steps Every organization is increasingly relying on APIs to build new applications, adopt modern architectures or create new experiences. In such transformation journeys, Apigee’s Pay-as-you-go pricing will provide flexibility for organizations to start small and scale seamlessly with their API management needs.To get started with Apigee’s Pay-as-you-go pricing go to console or try it for free hereCheck out our documentation and pricing calculator for further details on Apigee’s Pay-as-you-go pricing for API management. For comparison and other information, take a look at our pricing page.
Quelle: Google Cloud Platform

Building a Fleet of GKE clusters with ArgoCD

Organizations on a journey to containerize applications and run them on Kubernetes often reach a point where running a single cluster doesn’t meet their needs. One example, you want to bring your app closer to the users in a new regional market. Add a cluster to the new region and get the added benefit of increasing resiliency. Please read this multi-cluster use casesoverview  if you want to learn more about the benefits and tradeoffs involved.  ArgoCD and Fleets offer a great way to ease the management of multi-cluster environments by allowing you to define your clusters state based on labels abstracting away the focus from unique clusters to profiles of clusters that are easily replaced.This post shows you how to use ArgoCD and Argo Rollouts to automate the state of a Fleet of GKE clusters. This demo covers three potential journeys for a cluster operator. Add a new application cluster to the Fleet with zero touch beyond deploying the cluster and giving it a specific label. The new cluster should automatically install a baseline set of configurations for tooling and security along with any applications tied to the cluster label.Deploy a new application to the Fleet that automatically inherits baseline multi-tenant configurations for the team that develops and delivers the application, and applies Kubernetes RBAC policies to that team’s Identity Group.Progressively roll out a new version of an application across groups, or waves, of clusters with manual approval needed in between each wave.You can find the code used in this demo on GitHub.Configuring the ArgoCD Fleet ArchitectureArgoCD is a CNCF tool that provides GitOps continuous delivery for Kubernetes. ArgoCD’s UX/UI is one of its most valuable features. To preserve the UI/UX across a Fleet of clusters, use a hub and spoke architecture. In a hub and spoke design, you use a centralized GKE cluster to host ArgoCD (the ArgoCD cluster). You then add every GKE cluster that hosts applications as a Secret to the ArgoCD namespace  in the ArgoCD cluster. You assign specific labels to each application cluster to identify it. ArgoCD config repo objects are created for each Git repository containing Kubernetes configuration needed for your Fleet. ArgoCD’s sync agent continuously watches the config repo(s) defined in the ArgoCD applications and actuates those changes across the Fleet of application clusters based on the cluster labels that are in that cluster’s Secret in the ArgoCD namespace.Set up the underlying infrastructureBefore you start working with your application clusters, you need some foundational infrastructure. Follow the instructions in Fleet infra setup, which uses a Google-provided demo tool to set up your VPC, regional subnets, Pod and Service IP address ranges, and other underlying infrastructure. These steps also create the centralized ArgoCD cluster that’ll act as your control cluster.Configure the ArgoCD clusterWith the infrastructure set up, you can configure the centralized ArgoCD cluster with Managed Anthos Service Mesh (ASM), Multi Cluster Ingress (MCI), and other controlling components. Let’s take a moment to talk about why ASM and MCI are so important to your Fleet. MCI is going to provide better performance to all traffic getting routing into your cluster from an external client by giving you a single anycast IP in front of a global layer 7 load balancer that routes traffic to the GKE cluster in your Fleet that is closest to your clients. MCI also provides resiliency to regional failure. If your application is unreachable in the region closest to a client, they will be routed to the next closest region.Along with mTLS, layer 7 metrics for you apps, and a few other great features, ASM is going to provide you with a network that handles pod to pod traffic across your Fleet of GKE clusters. This means that your applications making calls to other applications within the cluster an automatically redirect to other cluster in your Fleet if the local call fails or has not endpoints. Follow the instructions in Fleet cluster setup. The command runs a script that installs ArgoCD, creates ApplicationSets for application cluster tooling and configuration, and logs you into ArgoCD. It also configures ArgoCD to synchronize with a private repository on GitHub.When you add a GKE application cluster as a Secret to the ArgoCD namespace, and give it the label `env: “multi-cluster-controller”`, the multi-cluster-controller ApplicationSet generates applications based on the subdirectories and files  in the multi-cluster-controllers folder.  For this demo, the folder contains all of the config necessary to setup Multi Cluster Ingress for the ASM Ingress Gateways that will be installed in each application cluster.When you add a GKE application cluster as a Secret to the ArgoCD namespace, and give it the label `env: “prod”`, the app-clusters-tooling application set generates applications for each subfolder in the app-clusters-config folder. For this demo, the app-clusters-config folder contains tooling needed for each application cluster. For example, the argo-rollouts folder contains the Argo Rollouts custom resource definitions that need to be installed across all application clusters.At this point, you have the following:Centralized ArgoCD cluster that syncs to a GitHub repository. Multi Cluster Ingress and multi cluster service objects that sync with the ArgoCD cluster.Multi Cluster Ingress and multi cluster Service controllers that configure the Google Cloud Load Balancer for each application cluster. The load balancer is only installed when the first application cluster gets added to the Fleet.Managed Anthos Service Mesh that watches Istio endpoints and objects across the Fleet and keeps Istio sidecars and Gateway objects updated.The following diagram summarizes this status:Connect an application cluster to the FleetWith the ArgoCD control cluster set up, you can create and promote new clusters to the Fleet. These clusters run your applications. In the previous step, you configured multi-cluster networking with Multi Cluster Ingress and Anthos Service Mesh. Adding a new cluster to the ArgoCD cluster as a Secret with the label `env=prod` ensures that the new cluster automatically gets the baseline tooling it needs, such as Anthos Service Mesh Gateways.To add any new cluster to ArgoCD, you add a Secret to the ArgoCD namespace in the control cluster. You can do this using the following methods:The `argocli add cluster` command, which automatically inserts a bearer token into the Secret that grants the control cluster `clusteradmin` permissions on the new application cluster.Connect Gateway and Fleet Workload Identity, which let you construct a Secret that has custom labels, such as labels to tell your ApplicationSets what to do, and configure ArgoCD to use a Google OAuth2 token to make authenticated API calls to the GKE control plane.When you add a new cluster to ArgoCD, you can also mark it as being part of a specific rollout wave, which you can leverage when you start progressive rollouts later in the demo.The following example Secret manifest shows a Connect Gateway authentication configuration and labels such as `env: prod` and `wave`:For the demo, you can use a Google-provided script to add an application cluster to your ArgoCD configuration. For instructions, refer to Promoting Application Clusters to the Fleet.You can use the ArgoCD web interface to see the progress of the automated tooling setup in the clusters, such as in the following example image:Add a new team application and a new clusterAt this point, you have an application cluster in the Fleet that’s ready to serve apps. To deploy an app to the cluster, you create the application configurations and push them to the ArgoCD config repository. ArgoCD notices the push and automatically deploys and configures the application to start serving traffic through the Anthos Service Mesh Gateway.For this demo, you can run a Google-provided script that creates a new application based on a template, in a new ArgoCD Team, `team-2`. For instructions, refer to Creating a new app from the app template. The new application creation also configures an application set for each progressive rollout wave, synced with a git branch for that wave. Since that application cluster is labeled as wave one and is the only application cluster deployed so far, you should only see one Argo application in the UI for the app that looks similar to this.If you `curl` the endpoint, the app responds with some metadata including the name of the Google Cloud zone in which it’s running:You can also add a new application cluster in a different Google Cloud zone, for higher availability. To do so, you create the cluster in the same VPC and add a new ArgoCD Secret with labels that match the existing ApplicationSets.For this demo, you can use a Google-provided script to do the following:Add a new cluster in a different zoneLabel the new cluster for wave two (the existing application cluster is labeled for wave one)Add the application-specific labels so that ArgoCD installs the baseline toolingDeploys another instance of the sample application in that clusterFor instructions, refer to Add another application cluster to the Fleet. After you run the script, you can check the ArgoCD web interface for the new cluster and application instance. The interface is similar to this:If you `curl` the application endpoint, the GKE cluster with the least latent path from the source of the curl serves the response. For example, curling from a Compute Engine instance in `us-west1` routes you to the `gke-std-west02` cluster.You can experiment with the latency-based routing by accessing the endpoint from machines in different geographical locations. At this point in the demo, you have the following:One application cluster labeled for wave oneOne application cluster labeled for wave twoA single Team with an app deployed on both application clustersA control cluster with ArgoCDA backing configuration repository for you to push new changesProgressively rollout apps across the FleetArgoCD rollouts are similar to Kubernetes Deployments, with some additional fields to control the rollout. You can use a rollout to progressively deploy new versions of apps across the Fleet, manually approving the rollout’s wave-based progress by merging the new version from the `wave-1` git branch to the `wave-2` git branch, and then into `main`. For this demo, you can use Google-provided scripts that do the following:Add a new application to both application clusters.Release a new application image version to the wave one cluster.Test the rolled out version for errors by gradually serving traffic from Pods with the new application image. Promote the rolled out version to the wave two cluster.Test the rolled out version.Promote the rolled out version as the new stable version in `main`. For instructions, refer to Rolling out a new version of an app. The following sample shows the fields that are unique to ArgoCD rollouts. The `strategy` field defines the rollout strategy to use. In this case, the strategy is canary, with two steps in the rollout. The application cluster rollout controller checks for image changes  to the rollout object and creates a new replica set with the updated image tag when you add a new image. The rollout controller then adjusts the Istio virtual service weight so that 20% of traffic to that cluster is routed to Pods that use the new image.Each step runs for 4 minutes and calls an analysis template before moving onto the next step. The following example analysis template uses the Prometheus provider to run a query to check the success rate of the canary version of the rollout. If the success rate is 95% or greater, the rollout moves on to the next step. If the success rate is less than 95%, the rollout controller  rolls the change back by setting the Istio virtual service weight to 100% for the Pods running the stable version of the image.After all the analysis steps are completed, the rollout controller labels the new application’s deployment as stable, sets the Istio virtual service 100% back to the stable step, and deletes the previous image version deployment. SummaryIn this post you have learned how ArgoCD and Argo Rollouts can be used to automate the state of a Fleet of GKE clusters. This automation abstracts away any uniques of a GKE cluster and allows you to promote and remove clusters as your needs change over time.  Here is a list of documents that will help you learn more about the services used to build this demo.Argo ApplicationSet controller: improved multi-cluster and multi-tenant support.Argo Rollouts: Kubernetes controller that provides advanced rollout capabilities such as blue-green and experimentation.Multi Cluster Ingress: map multiple GKE clusters to a single Google Cloud Load Balancer, with one cluster as the control point for the Ingress controller.Managed Anthos Service Mesh: centralized Google-managed control plane with features that spread your app across multiple clusters in the Fleet for high availability.Fleet Workload Identity: allow apps anywhere in your Fleet’s clusters that use Kubernetes service accounts to authenticate to Google Cloud APIs as IAM service accounts without needing to manage service account keys and other long-lived credentials.Connect Gateway: use the Google identity provider to authenticate to your cluster without needing VPNs, VPC Peering, or SSH tunnels.Related ArticleGoogle Kubernetes Engine: 7 years and 7 amazing benefitsHow you can benefit from 7 years of the most automated and scalable managed Kubernetes.Read Article
Quelle: Google Cloud Platform

Cloud CISO Perspectives: August 2022

Welcome to this month’s Cloud CISO Perspectives. This month, we’re focusing on the importance of vulnerability reward programs, also known as bug bounties. These programs for rewarding independent security researchers for reporting zero-day vulnerabilities to the software vendor first started appearing around 1995, and have since evolved into an integral part of the security landscape. Today, they can help organizations build more secure products and services. As I explain below, vulnerability reward programs also play a key role in digital transformation. As with all Cloud CISO Perspectives, the contents of this newsletter will continue to be posted to the Google Cloud blog. If you’re reading this on the website and you’d like to receive the email version, you can subscribe here.Why vulnerability rewards programs are vital to cloud servicesI’d like to revisit a Google Cloud highlight from June that I believe sheds some light onto an important aspect of how organizations build secure products, and build security into their business systems. On June 3, we announced the winners of the 2021 Google Cloud Vulnerability Rewards Program prize. This is the third year that Google Cloud has participated in the VRP. The top six prize winners scored a combined $313,337 for the vulnerabilities they found. An integral part of the competition is for the competitors to publish a public write-up of their vulnerability reports, which we hope encourages even more people to participate in open research into cloud security. (You can learn more about Google’s Vulnerability Rewards Program here.)Over the life of the program, we’ve increased the awards—a measure of the program’s success. And we’ve also increased the prize values in our companion Kubernetes Capture the Flag VRP. These increases benefit the research community, of course, and help us secure our products. But they also help develop a mature, resilient security ecosystem in which our internal security teams are indelibly connected to external, independent security researchers. This conclusion has been borne out by my own experience with VRPs, but also independent analysis. Researchers at the University of Houston and the Technical University of Munich concluded in a study of Chromium vulnerabilities published in 2021 that the diverse backgrounds and interests of external bug-hunters contributed to their ability to find different kinds of vulnerabilities. Specifically, they tracked down bugs in Chromium Stable releases and in user interface components. The researchers wrote that “external bug hunters provide security benefits by complementing internal security teams with diverse expertise and outside perspective, discovering different types of vulnerabilities.”Although organizations have used VRPs since the 1990s to help fix their software, and their use continues to grow in popularity, they still require forethought and planning. At the very least, an organization should have a dedicated, publicly-available and internally-managed email address for researchers to submit their reports and claims. More than anything else, researchers want to be able to communicate their security concerns to somebody who will take them seriously.That said, incoming vulnerability reports can set off klaxons if the preparations have not been put in place to properly manage them. A more mature VRP will triage incoming reports and have in place a more rigorous machinery which includes determining who will receive the reports, how the interactions with the researcher who filed the report will be handled, which engineering teams will be notified and involved, how the report will be verified as accurate and authentic, and how customers will be supported.There’s an opportunity for boards and organization leaders to take a more active role in kickstarting and guiding this process if their organization doesn’t have a VRP in place yet. Part of what makes VRPs so important is that they bring benefits beyond the obvious. They can help teams learn more, they can strengthen ties to the researcher community, they can provide feedback on updating internal processes, and they can create pathways to improve security and development team structures.Ultimately, the business case for a VRP program is simple. No matter how great you are at security, you still are going to have some vulnerabilities. You want those discovered as quickly as possible by people who will be incentivized to tell you. If you don’t, you run increasing risks that adversaries will either discover those vulnerabilities or acquire them from an illicit marketplace. As more organizations undergo their digital transformations, the need for VRPs will only increase. The web of interconnectedness between a company’s systems and the systems of its suppliers, partners, and customers will force them to expand the scope of their security concerns, so the most responsible behavior is for organizations to encourage their suppliers to adopt VRP programs.Google Cloud Security TalksSecurity Talks is our ongoing program to bring together experts from the Google Cloud security team, including the Google Cybersecurity Action Team and Office of the CISO, and the industry at large to share information on our latest security products, innovations, and best practices. Our latest Security Talks, on Aug. 31, will focus on practitioner needs and how to use our products. Sign up here. Google Cybersecurity Action Team highlightsHere are the latest updates, products, services and resources from our security teams this month: SecurityHow Google Cloud blocked the largest Layer 7 DDoS attack to date: On June 1, a Google Cloud Armor customer was targeted with a series of HTTPS DDoS attacks which peaked at 46 million requests per second. This is the largest Layer 7 DDoS reported to date—at least 76% larger than the previously reported record. Here’s how we stopped it. “Deception at scale”—VirusTotal’s latest report: VirusTotal’s most recent report on the state of malware explores how malicious hackers change up their malware techniques to bypass defenses and make social engineering attacks more effective.  Read more.First-to-market Virtual Machine Threat Detection now generally available: Our unique Virtual Machine Threat Detection (VMTD) in Security Command Center is now generally available for all Google Cloud customers. Launched six months ago in public preview, VMTD is invisible to adversaries and draws on expertise from Google’s Threat Analysis Group and Google Cloud Threat Intelligence. Read more. How autonomic data security can help define cloud’s future: As data usage has undergone drastic expansion and changes in the past five years, so have your business needs for data. Google Cloud is positioned uniquely to define and lead the effort to adopt a modern approach to data security. We contend that the optimal way forward is with autonomic data security. Here’s why.How CISOs need to adapt their mental models for cloud security: Successful cloud security transformations can help better prepare CISOs for threats today, tomorrow, and beyond, but they require more than just a blueprint and a set of projects. CISOs and cybersecurity team leaders need to envision a new set of mental models for thinking about security, one that will require you to map your current security knowledge to cloud realities. Here’s why. How to help ensure smooth shift handoffs in security operations: Without proper planning, SOC shift-handoffs can create knowledge gaps between team members. Fortunately, those gaps are not inevitable. Here’s three ways to avoid them. Five must-know security and compliance features in Cloud Logging: As enterprise and public sector cloud adoption continues to accelerate, having an accurate picture of who did what in your cloud environment is important for security and compliance purposes. Here are five must-know Cloud Logging security and compliance features (including three new ones launched this year) that can help customers improve their security audits. Read more.Google Cloud Certificate Authority Service now supports on-premises Windows workloads: Organizations who have adopted cloud-based CAs increasingly want to extend the capabilities and value of their CA to their on-premises environments. They can now deploy a private CA through Google Cloud CAS along with a partner solution that simplifies, manages, and automates the digital certificate operations in on-prem use cases such as issuing certificates to routers, printers, and users. Read more. Easier de-identification of Cloud Storage data: Many organizations require effective processes and techniques for removing or obfuscating certain sensitive information in the data that they store, a process known as “de-identification.” We’ve now released a new action for Cloud Storage inspection jobs that makes this process easier. Read more. Introducing Google Cloud and Google Workspace support for multiple Identity providers with Single Sign-On: Google has long provided customers with a choice of digital identity providers. For more than a decade, we have supported SSO via the SAML protocol. Currently, Google Cloud customers can enable a single identity provider for their users with the SAML 2.0 protocol. This release significantly enhances our SSO capabilities by supporting multiple SAML-based identity providers instead of just one. Read more. Curated detections come to Chronicle SecOps Suite: A critical component of any security operations team’s job is to deliver high-fidelity detections of potential threats across the breadth of adversary tactics. Today, we are putting the power of Google’s intelligence in the hands of security operations teams with high quality, actionable, curated detections built by our Google Cloud Threat Intelligence team. Read more. Google Cloud’s Managed Microsoft Active Directory gets on-demand backup, schema extension support: We’ve added schema extension support and on-demand backups to our Managed Microsoft Active Directory to make it easier for customers to integrate with applications that rely on AD. Read more.Securing apps using Anthos Service Mesh: Our Anthos Service Mesh can help maintain a high level of security across numerous apps and services with minimal operational overhead, all while providing service owners granular traffic control. Here’s how it works. Our Security Voices blogging initiative highlights blogs from a diverse group of Google Cloud’s security professionals. Here, Jaffa Edwards explains how preventive security controls, also known as security “guardrails,” can help developers prevent misconfigurations before they can be exploited. Read more.Industry updatesHow Vulnerability Exploitability eXchanges can help healthcare prioritize cybersecurity risk: In our latest blog on healthcare and cybersecurity resiliency, we discuss how a VEX can help bolster SBOM and SLSA with vital information for making risk-assessment decisions in healthcare organizations—and beyond. Read more. MITRE and Google Cloud collaborate on cloud analytics: How can the cybersecurity industry improve its analysis of the already-tremendous and growing volumes of security data in order to better stop the dynamic threats we face? We’re excited to announce the release of the Cloud Analytics project by the MITRE Engenuity Center for Threat-Informed Defense, and sponsored by Google Cloud and several other industry collaborators. Read more. Compliance & ControlsUsing data advocacy to close the consumer privacy trust gap: As consumer data privacy regulations tighten and the end of third-party cookies looms, organizations of all sizes may be looking to carve a path toward consent-positive, privacy-centric ways of working. Organizations must begin to treat consumer data privacy as a pillar of their business. One way to do this is by implementing a cross-functional data advocacy panel. Read more.How to avoid cloud misconfigurations and move towards continuous compliance: Modern application security tools should be fully automated, largely invisible to developers, and minimize friction within the DevOps pipeline. Infrastructure continuous compliance can be achieved thanks to Google Cloud’s open and extensible architecture, which uses Security Command Center and open source solutions. Here’s how.Helping European education providers navigate privacy assessments: Navigating complex DPIA requirements under GDPR can be challenging for many of our customers, and while only customers, as controllers, can complete DPIAs, we are here to help meet these compliance obligations with our Cloud DPIA Resource Center. Read more. Tips for security teams to shareAs I noted in July’s newsletter, we published four helpful guides that month on Google Cloud’s security architecture. These explainers by our lead developer advocate Priyanka Vergadia are ready-made to share with IT colleagues, and come with colorful illustrations that break down how our security works. This month, we added two more. Make the most of your cloud deployment with Active Assist: This guide walks you through our Active Assist feature, which can help streamline information from your workloads’ usage, logs, and resource configuration, and then uses machine learning and business logic to help optimize deployments in exactly those areas that make the cloud compelling: cost, sustainability, performance, reliability, manageability, and security. Read more. Zero Trust and BeyondCorp: In this primer, we focus on how the need to mitigate the security risks created by implicitly trusting any part of a system has led to the rise of the Zero Trust security model. Read more.Google Cloud Security PodcastsWe launched in February 2021 a new weekly podcast focusing on Cloud Security. Hosts Anton Chuvakin and Timothy Peacock chat with cybersecurity experts about the most important and challenging topics facing the industry today. This month, they discussed:Demystifying data sovereignty at Google Cloud: What is data sovereignty, why it matters, and how it will play a growing role in cloud technology, with Google’s C.J. Johnson. Listen here.A CISO walks into a cloud: Frustrations, successes, lessons, and risk, with David Stone, staff consultant at our Office of the CISO. Listen here. How to modernize data security with the Autonomic Data Security approach, with John Stone, staff consultant at our Office of the CISO. Listen here.What changes and what doesn’t when SOC meets cloud, with Gorka Sadowski, chief strategy officer at Exabeam. Listen here.Explore the magic (and operational realities) of SOAR, with Cyrus Robinson, SOC Director and IR Team lead at Ingalls Information Security. Listen here.To have our Cloud CISO Perspectives post delivered every month to your inbox, sign up for our newsletter. We’ll be back next month with more security-related updates.Related ArticleCloud CISO Perspectives: July 2022Google Cloud CISO Phil Venables shares his thoughts on the important role and challenges of including cybersecurity in the boardroom, alo…Read Article
Quelle: Google Cloud Platform

Introducing Vertical Autoscaling in streaming Dataflow Prime jobs

Dataflow has provided a number of capabilities to improve utilization and efficiency by automatically provisioning and scaling resources for your job. The following are some of the examples:Horizontal Autoscaling that automatically scales the number of workers.Streaming Engine, which decouples storage from the workers. This also gives workers access to unbounded storage and more responsive Horizontal Autoscaling.Dynamic work rebalancing that splits work across available workers based on work progress. Building on this solid and differentiated foundation, we recently launched Dataflow Prime, a new next generation serverless, no-ops, autotuning platform for your data processing needs on Google Cloud. Dataflow Prime introduces a new industry-first resource optimization technology, Vertical Autoscaling, which automatically scales worker memory in order to remove the need to do manual tuning of worker configuration. With Vertical Autoscaling, Dataflow Prime automatically determines the right worker configuration for your job. Current user challenges With Dataflow, you write your data processing logic using the Apache Beam SDK or Dataflow Templates and let Dataflow handle the optimization, execution and scalability of your pipeline. While in many cases your pipeline executes well, for some cases you have to manually select the right resources like memory for best performance and cost. For many users, this is a time consuming trial and error process and a single worker configuration is unlikely to be optimal for the pipeline. In addition, there was the risk of static configurations becoming outdated when data processing requirements changed. We have designed Vertical Autoscaling to solve these challenges and allow you to focus on your application and business logic.   How does Vertical Autoscaling work?Vertical Autoscaling observes out of memory (OOM) events and memory usage of your streaming pipeline over time and triggers memory scaling based on this. This makes your pipeline resilient to out of memory errors without any manual intervention.With Vertical Autoscaling, if there is high memory utilization, all workers in your job are replaced with workers with larger memory capacity. In the following illustration we see that workers 1, 2, and 3 have high memory utilization and a capacity of 4GB. After Vertical Autoscaling, all workers have a memory capacity of 5 GB, which gives them sufficient memory headroom.This process is iterative, and it can take up to a few minutes to replace the workers.Similarly, if there is low memory usage, Vertical Autoscaling downscales the workers to lower memory capacity, thus improving utilization and saving cost. It relies on historical usage data per pipeline to know when it is safe to scale down, prioritizing pipeline stability. You may observe a long period of time (12 hours or more) where no downscaling occurs after a spike in memory utilization. Vertical Autoscaling takes a conservative approach to downscaling in order to keep pipelines processing with minimal disruption.Things to know about Vertical Autoscaling How does Vertical Autoscaling impact my job?As the workers are replaced, you may observe a temporary drop in throughput, but impact to a running pipeline (i.e. backlog, watermark, throughput metrics) will not be significantly different from a Horizontal Autoscaling event.Horizontal Autoscaling is disabled during and up to 10 minutes after Vertical Autoscaling.As with horizontal scaling, some backlog may accumulate during the scaling process, if this backlog cannot be cleared in a timely fashion, horizontal scaling may occur to clear that backlog.Does Vertical Autoscaling remove all OOMs?It is important to note that Vertical Autoscaling is designed to react to OOMs and high memory usage, but cannot necessarily prevent OOMs, especially if there is a fast spike in memory usage on a worker which results in an OOM.When OOMs occur, Vertical Autoscaling automatically detects them and resize worker memory to address issues. As a consequence, you will see a few OOM errors in the worker logs but these can be ignored if those are followed by upscale events.It is also important to note that some OOMs may be happen as a result  of downscale events where Dataflow reduced the amount of memory because of underutilization. In such cases, Dataflow will automatically upsize if it detects OOMs. Again, it is safe to ignore these OOM messages if they are followed by upscale events.If the OOM messages are not followed by an upscale event, you may have hit the memory scaling limit. In this case you may need to optimize your pipeline’s memory usage or use resource hints.If you see OOM messages continuously and have not observed a job message indicating you have hit the memory scaling limit, please contact the support team. Note that if OOMs occur very rarely (e.g. once every few hours per pipeline), Vertical Autoscaling may choose to not scale up the workers to avoid introducing additional disruption.How to enable Vertical Autoscaling ?Vertical Autoscaling is only available for Dataflow Prime jobs. See the instructions to launch Dataflow Prime jobs and how to enable Vertical Autoscaling. You don’t have to make any code changes to run your existing Apache Beam pipeline on Dataflow Prime. Additionally, you don’t have to specify the worker type when launching a Dataflow Prime job. However if you want to control the initial worker’s resource configuration you can use resource hints.You can confirm if Vertical Autoscaling is running on your pipeline by looking for the following job log:Vertical Autoscaling is enabled. This pipeline is receiving recommendations for resources allocated per worker. How to monitor Vertical Autoscaling ?Whenever Vertical Autoscaling updates workers with more or less memory, the following job logs are generated in Cloud Logging: Vertical Autoscaling is enabled. This pipeline is receiving recommendations for resources allocated per worker. Vertical Autoscaling update triggered to change per worker memory limit for pool from X GiB to Y GiB. You can read more about these logs in this section.Additionally you can visually monitor Vertical Autoscaling by looking at worker capacity under the ‘Max worker memory utilization’ chart in Dataflow metrics UI.Following is a chart for a Dataflow worker that Vertically Autoscaled. We see three Vertical Autoscaling events for this job. Whenever the memory used was close to memory capacity, Vertical Autoscaling triggered and scaled up the worker memory capacity.SummaryTryVertical Autoscaling for your streaming jobs on Dataflow Prime for improved resource optimization and cost savings.There is no code change required to run your existing Apache Beam pipeline on Dataflow Prime.There is no additional cost associated with using Vertical Autoscaling. Dataflow Prime jobs continue to be billed based on the resources it consumes.Related ArticleThe next generation of Dataflow: Dataflow Prime, Dataflow Go, and Dataflow MLDataflow is GCP’s Cloud Native way for all data processing workloads, powered by the universal batch and streaming model of Apache Beam.Read Article
Quelle: Google Cloud Platform

Introducing on-demand backup, schema extension support for Google Cloud’s Managed Microsoft AD

Managed Service for Microsoft Active Directory (Managed Microsoft AD) is a Google Cloud service that offers highly available, hardened Microsoft Active Directory running on Windows virtual machines. We recently added on-demand backup and schema extension capabilities that can help Google Cloud users more easily and effectively manage AD tasks. Managed Microsoft AD is a fully managed service with automated AD server updates, maintenance, and security configuration, and needs no hardware management or patching. The service is constantly evolving, adding new capabilities to effectively manage your cloud-based, AD-dependent workloads. Here’s a closer look at the benefits for Google Cloud users of the new on-demand backup and schema extension capabilities.Flexibility to manage your AD domain with on-demand backup and restoreManaged Microsoft AD already offers scheduled backups which are taken automatically every 12 hours. Now with on-demand backup and restore, customers will have the ability to create checkpoints (snapshots) at any point in time and restore back to that state when needed. The new on-demand backup and restore functionality is now generally available in addition to the scheduled backups. This functionality can provide flexibility for customers to initiate backup and recovery based on their unique needs. Here are two scenarios where on-demand backup and recovery can be used:Critical domain changes now can be done anytime without aligning to the next backup schedule. Users can restore to a point back in time from backups without having to raise a support request.With this release, users can create up to five on-demand backups. Managed Microsoft AD APIs also offer management functionalities for backups that includes listing of all backups (both on-demand and scheduled,) restoring to a selected backup, updating labels, and deleting a backup. All these capabilities help users to effectively manage their backup administrative tasks. Power application integrations with Schema Extension supportNote: Schema Extension feature is in public preview and covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Active Directory (AD) relies on schema to organize and store the directory data. The AD schema contains a formal definition of every attribute and class that can exist in an Active Directory object. When you create a Managed Microsoft AD instance, it creates a default schema on the domain controller as well. However, there can be a situation where you want to customize the classes or attributes. Such a need arises when you have applications that require new types of information to be stored in Active Directory (e.g., to support single sign-on capabilities). Managed Microsoft AD now supports schema extension and enables modification of the existing schema to customize attributes via API using an LDAP Data Interchange Format (LDIF) file. The following LDIF change types are supported: add, modify, modrdn and moddn. It is generally recommended to do a domain backup before schema changes are applied. To simplify this, Managed Microsoft AD initiates a backup every time schema changes are triggered. This schema extension support enables additional context for users and for integrating with applications that are dependent on specific classes or attributes.Use case: Schema extension for LAPSYou can store and rotate the local account passwords of domain-joined computers in AD using Local Administrator Password Solution (LAPS), a Microsoft tool for password management. Any device that LAPS is deployed to can randomize the local administrator password, store that password in Active Directory, and then change that password on a set schedule. For LAPS to work with Active Directory, it needs the schema to be extended for storing the required attributes. For this use case, we assume that you have already installed LAPS and have your Managed Microsoft AD up and running. LAPS requires the following two additional attributes:ms-Mcs-AdmPwd – This attribute stores the local administrator passwordms-Mcs-AdmPwdExpirationTime – This attribute stores the expiration time of administrator passwordLet’s now look at how to add the required attributes using the Managed Microsoft AD schema extension feature.Step 1: Prepare an LDIF file to add ms-Mcs-AdmPwd and ms-Mcs-AdmPwdExpirationTime attributes.code_block[StructValue([(u’code’, u’dn: CN=ms-Mcs-AdmPwd,CN=Schema,CN=Configuration,dc=example,dc=comrnchangetype: addrnobjectClass: attributeSchemarnldapDisplayName: ms-Mcs-AdmPwdrnadminDisplayName: ms-Mcs-AdmPwdrnadminDescription: LAPS PasswordrnattributeId: 1.2.840.113556.8000.9999.2.2rnattributeSyntax: 2.5.5.5rnoMSyntax: 19rnisSingleValued: TRUErnsystemOnly: FALSErnsearchFlags: 904rnschemaIdGuid:: 64e85e0a-f479-4206-880d-ecbf73e2babbrnrnrndn: CN=ms-Mcs-AdmPwdExpirationTime,CN=Schema,CN=Configuration,dc=example,dc=comrnchangetype: AddrnobjectClass: attributeSchemarnldapDisplayName: ms-Mcs-AdmPwdExpirationTimernadminDisplayName: ms-Mcs-AdmPwdExpirationTimernadminDescription: LAPS Password Expiration TimernattributeId: 1.2.840.113556.8000.9999.2.3rnattributeSyntax: 2.5.5.6rnoMSyntax: 65rnisSingleValued: TRUErnsystemOnly: FALSErnsearchFlags: 0rnschemaIdGuid:: b3fea135-c39a-4169-aec7-c618cc8cb6ffrnrndn:rnchangetype: modifyrnadd: schemaUpdateNowrnschemaUpdateNow: 1′), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b788850>)])]Step 2: Login as a delegated administrator to your VM hosted in Google Cloud that was domain-joined with Managed Microsoft AD.Step 3: Extend the schema by running the following gCloud CLI command:code_block[StructValue([(u’code’, u’gcloud beta active-directory domains extend-schema DOMAIN_NAME –ldif-file=LDIF_FILE_PATH –description=u201dSample descriptionu201d –project=PROJECT_ID’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b4f0d50>)])]Managed Microsoft AD creates a backup automatically when you initiate schema extension. You can use this backup to perform an authoritative restore, which returns the domain to a previous point before addition of these attributes. Step 4: To verify the schema changes, run the following command in Windows PowerShell:code_block[StructValue([(u’code’, u”get-adobject -Identity ‘cn=ATTRIBUTE,cn=Schema,cn=Configuration,dc=example,dc=com -Properties *”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b4f0410>)])]The Managed Microsoft AD schema is now extended with the required attributes for configuring LAPS. You can now proceed with the rest of the LAPS setup as usual, including password settings, access permissions, and GPO configuration.These new features make it now easier to integrate applications with your Managed Microsoft AD and provide flexibility for operations like backup and restore. Here are additional resources where you can learn more about Managed AD and these new features.Managed Service for Microsoft AD documentationBackup and restore a domain in Managed Microsoft ADIntroduction to schema extension in Managed Microsoft ADExtend the schema in a Managed Microsoft ADRelated ArticleAnnouncing support for on-premises Windows workloads with Certificate Authority ServiceTo mitigate the challenges in managing and migrating digital certificates, Google Cloud is debuting a new technology partnership with CA …Read Article
Quelle: Google Cloud Platform

How to avoid cloud misconfigurations and move towards continuous compliance

Security is often seen as a zero-sum game between “go fast” or “stay secure.” We would like to challenge this school of thought and introduce a framework to change that paradigm to a “win-win game,” so you can do both—“go fast” and “stay secure.” Historically, application security tools have been implemented much like a gate at a parking lot. The parking lot has perimeter-based ingress and egress boom gates. The boom gates let one car through at a time, and vehicles often are backed up at the gates during busy hours. However, there are few controls once you get inside. You can access nearly any space on any level and easily move between levels.When you apply this analogy to application development, AppSec tools are often implemented as “toll gates” within waterfall-native workflows. Developers are required to get in line, submit to a security scan, and wait to see the results. When the results are produced, developers spend significant time and energy investigating red flags raised by security. This process is slow and, not surprisingly, not very popular with developers. It’s why they often view traditional security programs as inhibitors to innovation.Guardrails not gatesWe suggest a workflow that’s less like a parking lot gate and more like a freeway with common-sense safety measures. Freeways have directive rules for all users. Speed limits, single direction of travel, and mandatory speed reduction zones when exiting contribute to freeway safety. Some freeways implement preventative measures based on these rules, such as physical walls dividing opposite flows of traffic and protective guardrails to reduce collisions and keep vehicles from veering off the road. While driving on a freeway comes with its own complications, there are no boom-style gates blocking your path. Following the same directive rules, there are detective and responsive controls, such as speed detectors, cameras, signs reminding drivers which direction they are going, and how fast they are traveling. Some freeways have deployed rumble strips to remind a dozing driver to stay in their lane. Applying lessons from freeways to application development and compliance in the cloud represents the perfect opportunity to build software more securely.Modern application security tools should be fully automated, largely invisible to developers, and minimize friction within the DevOps pipeline. To do this, these security tools should work the way developers want to work. Security controls should integrate into the development lifecycle early and everywhere. These controls should live within the developer’s preferred tools and create rapid feedback loops so mistakes can be remediated as soon as possible.A typical compliance cycle looks like this:Here, we highlight the gap between the desired state and the actual state that becomes problematic when audit times come. This increases the overall cost of the audit and the time spent in generating the evidence of controls.Instead, this is what we need.We need the actual state to track the desired state continuously. We need continuous preventative controls to stop insecure resources from being introduced. We need detective controls to find non-compliant resources promptly and constantly. We need responsive controls to fix non-compliant resources automatically. In all, we need continuous compliance.Infrastructure continuous compliance reference architectureHow do we get started with continuous compliance? Here is the reference architecture that enables you to develop this capability.The architecture is centered on building a close-loop of directive, preventative, detective and responsive controls. It is also open and extensible. Although we reference Google Cloud architectures in this blog, you can use them for other cloud service platforms or even on-premise. The National Institute of Standards and Technology’s Open Security Controls Assessment Language (OSCAL) is a helpful resource to express your control library in a machine-readable format. OSCAL can allow organizations to define a set of security and privacy requirements, which are represented as controls, which then can be grouped together as a control catalog. Organizations can use these catalogs to establish security and privacy control baselines through a process that may aggregate and tailor controls from multiple source catalogs. Using the OSCAL profile model to express a baseline makes the mappings between the control catalog and the profile explicit and machine-readable.Directive controlsThe starting point of the close-loop is the directive and harmonized controls. Next, you should have control mappings rationalized to the technical controls against your compliance requirements. These requirements can come from various sources, such as the threat landscape of your industry, your internal security policies and standards, your external regulatory compliance, and industry best practice frameworks. Control mappings will form a Technical Control Library. The library is a dataset mapping out harmonized controls to requirements written in different compliance frameworks. The control mapping justifies the security controls. It builds the linkage between security and compliance and helps you reduce your compliance audit cost. This dataset should be a living document. An easy first step in building such as library is to begin with the CIS Google Cloud Platform Foundation Benchmark. The benchmark is lightweight and it constitutes foundational security any entity should get right on Google Cloud. In addition, Security Command Center Premium’s  Security Health Analytics can help you to monitor your Google Cloud environment against these benchmarks  on a continuous basis across all the projects within your organization. The Technical Control Library will guide the rest of the closed-loop. For every directive control, you should have corresponding preventative control to stop non-compliant resources from being deployed. You should have the detective control to look over the entire environment seeking non-compliant resources. And you should have the responsive control remedying non-compliant resources automatically or kicking off responsive workflow with your Security Operations function. Finally, every policy evaluation point should have a feedback loop to the engineers. A prompt and meaningful feedback loop provides a better engineering experience and increases development velocity in the short run. These feedback loops will breed good behaviors to write better and more secure code in the long run.Preventive controlsAlmost every action on the Google Cloud is an API call, such as when creating, configuring, or deleting resources, so preventative controls are all about API call constraints. There are different wrappers for these API calls, including Infrastructure-as-Code (IaC) solutions such as Terraform or Google Cloud Deployment Manager, the Cloud Console interface, Cloud Shell SDK, Python, or GO SDK. As with any other application code deployment, the IaC solutions should use a Continuous Integration (CI) solution. On the CI, you could orchestrate IaC constraints, similar to writing unit tests for application code. Since all IaC solutions come in or can be converted to JSON format, you can use Open Policy Agent (OPA) as the IaC constraint solution. OPA’s Rego policy language is declarative and flexible, which allows you to construct almost any policy in Rego. For the input sources that are not IaC, you could fall back to the organization policies and IAM as these two controls have the closest proximity to Google Cloud. That said, it’s considered a best practice to restrict non-IaC inputs for higher environments such as production-like or production, so you could codify your infrastructure, apply controls and workflows in the source repository. Detective and responsive controlsEven if you’ve nailed the preventive controls, and the cloud environment is sterile, we still need detective and responsive controls. Here’s why. For one, not all the controls can be safely implemented as preventative controls in the real world. For instance, we may not fail all the Google Compute Engine deployments at the CI if these VMs have external IP addresses because external IP addresses may be required for a specific software or use cases. Another reason is that we want to produce time-stamped compliance status for audit purposes. Taking the CIS compliance as an example, we could have enforced all the CIS check on the CI and set IaC as the only deployment source for cloud infrastructure. However, we will still need to demonstrate the runtime CIS compliance report using Security Command Center. Security responsive controls are not limited to remediation actions. They can also take the form of notifications via email, messaging tools, or integration with ITSM systems. If you use Terraform to deploy the infrastructure and use Cloud Function for auto-remediation, you need to pay attention to the Terraform state. Since auto-remediation actions performed by Cloud Function are not recorded in the Terraform state file, you will need to inform the engineers to update the source Terraform code.The futureThe fact that manual processes around security and compliance don’t scale points to automation as the next enabler. The economics of automation require a systemic discipline and holistic enterprise-wide approach to regulatory compliance and cloud risk management. By defining a data model of the compliance process, the aforementioned OSCAL represents a game-changer for automation in risk management and regulatory compliance. While we realize that adopting “as code” practices is a long-term investment for most of our customers, Risk and Compliance as Code (RCaC) has a number of building blocks to get you started. By adopting the RCaC tenets you shift towards codified policies and infrastructure for a secure cloud transformation. Stay tuned as we introduce exciting new capabilities and features to Google Cloud Risk and Compliance as Code in the months to come.
Quelle: Google Cloud Platform

Spatial Clustering on BigQuery – Best Practices

Most data analysts are familiar with the concept of organizing data into clusters so that it can be queried faster and at a lower cost. The user behavior dictates how the dataset should be clustered: for example, when a user seeks to analyze or visualize geospatial data (a.k.a location data), it is most efficient to cluster on a geospatial column. This practice is known as spatial clustering, and in this blog, we will share best practices for implementing it in BigQuery (hint — let BigQuery do it for you). BigQuery is a petabyte-scale data warehouse that has many geospatialcapabilities and functions. In the following sections, we will describe how BigQuery does spatial clustering out of the box using the S2 indexing system. We will also touch on how to use other spatial indexes like H3 and geohash, and compare the cost savings of different approaches. How BigQuery does spatial clustering under the hoodClustering ensures that blocks of data with similar values are colocated in storage, which means that the data is easier to retrieve at query time. It also sorts the blocks of data, so that only the necessary blocks need to be scanned, which reduces cost and processing time. In geospatial terms, this means that when you’re querying a particular region, only the rows within or close to that region are scanned, rather than the whole globe.All of the optimizations described above will occur automatically in BigQuery if you cluster your tables on a GEOGRAPHY column. It’s as easy as typing CLUSTER BY [GEOGRAPHY] when creating the table. Only predicate functions (e.g. ST_Intersects, ST_DWithin) leverage clustering, with the exception of ST_DISJOINT. It should also be noted that while BigQuery supports partitioning and clustering on a variety of fields, only clustering is supported on a geospatial field. This is because geometries can be large and could span across partitions, no matter how BigQuery chooses to partition the space. Finally, cluster sizes will range from 100MB to 1GB, so clustering on a table smaller than 100MB will provide no benefit.When writing to a table that is clustered by GEOGRAPHY, BigQuery shards the data into spatially-compact blocks. For each block, BigQuery computes a bit of metadata called an S2 covering that includes the spatial area of the data contained within. When querying a geography-clustered table using spatial predicates, BigQuery reads the covering and evaluates whether a particular covering can satisfy the filter. BigQuery then prunes the blocks that cannot satisfy the filter. Users are only charged for data from remaining blocks. Note that S2 coverings can overlap, as it is often impossible to divide data into non-overlapping regions. Fundamentally, BigQuery is using the S2 index to map a geometry into a 64-bit integer, then BigQuery clusters on that integer using existing integer-based clustering mechanisms. In the past, customers have manually implemented an S2 indexing system in BigQuery. This was done prior to BigQuery’s native support of spatial clustering via S2. Using BigQuery’s native clustering resulted in a large performance increase, not to mention the added simplicity of not having to manage your own S2 indexes.Alternative Spatial IndexesSpatial clustering utilizes a spatial indexing system, or hierarchy, to organize the stored data. The purpose of all spatial indices is to represent this globe we call Earth in numerical terms, allowing us to define a location as a geometric object like a point, polygon or line. There are dozens of spatial indexes, and most databases implement them in their own unique way. Although BigQuery natively uses S2 cells for clustering, other indexes can be manually implemented, such as H3, geohash, or quadkeys. The examples below will involve the following spatial indexes:S2:  The S2 system represents geospatial data as cells on a three dimensional sphere. It is used by Google Maps.uses quadrilaterals, which are more efficient than hexagonsHigher precision than H3 or geohashingH3:  The H3 system represents geospatial data on overlapping hexagonal grids.Hexagons are more visually appealing Convolutions and smoothing algorithms are more efficient than S2Geohash – Geohash is a public domain system that represents geospatial data on a curved grid.  Length of the Geohash id determines the spatial precisionFairly poor spatial locality, so clustering does not work as wellSpatial clustering in BQ — S2 vs. GeohashIn most cases for analysis, BigQuery’s built-in spatial clustering will give the best performance with the least effort. But if the data is queried according to other attributes, e.g. by geohash box, a custom indexing is necessary. The method of querying the spatial indexes has implications on the performance, as is illustrated in the example below. ExampleFirst, you will create a table with random points in longitude and latitude. Use the BigQuery function st_geohash to generate a geohash id for each point.code_block[StructValue([(u’code’, u’drop table if exists tmp.points;rn rncreate or replace table tmp.tenkrows asrnselect x from unnest(generate_array(1, 10000)) x;rn rncreate or replace table tmp.pointsrncluster by pointrnasrnwith pts as (rn select st_geogpoint(rand() * 360 – 180, rand() * 180 – 90) pointrn from tmp.tenkrows a, tmp.tenkrows brn)rnselect st_geohash(point) gh, pts.pointrnfrom pts’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed25c361210>)])]Use the st_geogpoint function to transform the latitude and longitude into a GEOGRAPHY, BigQuery’s native geospatial type, which uses S2 cells as the index. Select a collection of around 3,000 points. This should cost around 25MB. If you run the same query on an unclustered table, it would cost 5.77GB (the full table size).code_block[StructValue([(u’code’, u’select * from tmp.pointsrnwhere st_dwithin(st_geogpoint(1, 2), point, 10000)’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed25eb11610>)])]Now you will query by geohash id. BigQuery’s ability to leverage the spatial clustering will depend on whether the BQ SAT solver can prove the cluster of data can be pruned. The queries below are both leveraging the geospatial clustering, costing only 340 MB. Note that if we had clustered the table by the ‘gh’ field (ie geohash id), these queries would cost the same as the one above, around 25MB.code_block[StructValue([(u’code’, u”select * from tmp.pointsrnwhere starts_with(gh, ‘bbb’)rn rnselect * from tmp.pointsrnwhere gh between ‘bbb’ and ‘bbb~'”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed25d0e5ad0>)])]The query below is much less efficient, costing 5.77GB, a full scan of the table. BigQuery cannot prove this condition fails based on the min/max values of the cluster so it must scan the entire table.code_block[StructValue([(u’code’, u”select * from tmp.pointsrnwhere left(gh, 3) = ‘bbb'”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed25d0e55d0>)])]As the examples show, the least costly querying option is to use the indexing consistent with the query method — native S2 indexing when querying by geography, string indexing when querying by geohash. When using geohashing, avoid left() or right() functions, as it will cause BigQuery to scan the entire table.Spatial clustering in BQ with H3One may also find themselves in a situation where they need to use H3 as a spatial index in BigQuery. It is still possible to leverage the performance benefits of clustering, but as with geohashing, it is important to avoid certain patterns. Suppose you have a huge table of geography points indexed by H3 cell ID at level 15, which you’ve clustered by H3_index (note: these functions are supported through the Carto Spatial Extension for BigQuery). You want to find all the points that belong to lower resolution cells, e.g. at level 7. You might write a query like this:code_block[StructValue([(u’code’, u’select * from pointsrn where H3_ToParent(h3_index, 7) = @parent_cell_id’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed24322d110>)])]Where H3_ToParent is a custom function that computes the parent cell ID from a higher resolution index. Since you’ve clustered by the H3 index, you might expect a lower cost, however this query will scan the entire table. This happens because H3_ToParent involves bit operations, and is too complex for the BigQuery query analyser to understand how the query’s result is related to cluster boundaries. What you should do instead is give BigQuery the range of the H3 cell IDs at the level that the geographies are indexed, like the following example:code_block[StructValue([(u’code’, u’select * from pointsrn where h3_index BETWEEN H3_CellRangeStart(@parent_cell_id, 15)rn AND H3_CellRangeEnd(@parent_cell_id, 15)’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3ed2431ffa10>)])]Where H3_CellRangeStart and H3_CellRangeEnd are custom functions that map the lower-resolution parent ID to the appropriate start and end IDs of the higher resolution cells. Now BigQuery will be able to figure out relevant clusters, reducing the cost and improving the performance of the query.What’s Next?Spatial clustering is a complex topic that requires specialized knowledge to implement. Using BiqQuery’s native spatial clustering will take most of the work out of your hands. With your geospatial data in BigQuery, you can do amazing spatial analyses like querying the stars, even on large datasets. You can also use BigQuery as a backend for a geospatial application, such as an application that allows customers to explore the climate risk of their assets. Using spatial clustering, and querying your clusters correctly will ensure you get the best performance at the lowest cost. Acknowledgments: Thanks to Eric Engle and Travis Webb for their help with this post.Related ArticleQuerying the Stars with BigQuery GISDr. Ross Thomson explains how you can use BigQuery-GIS to analyze astronomy datasets, in a similar manner to analyzing ground-based map d…Read Article
Quelle: Google Cloud Platform

Google Kubernetes Engine: 7 years and 7 amazing benefits

Today, as we celebrate seven years of general availability of the most automated and scalable managed Kubernetes, Google Kubernetes Engine (GKE), we present seven of the common ways that GKE helps customers do amazing things.Accelerates productivity of developersDeveloper time is at a premium. GKE provides a rich set of integrated tools to help you ship faster and more often. The practice of continuous integration (CI) allows developers to frequently integrate all their code changes back into a main branch, exposing failures faster by revealing issues as early as possible in the process. A CI pipeline typically produces an artifact that you can deploy in later stages of the deployment process with continuous delivery (CD). CD lets you release code at any time.The ecosystem of developer tools for GKE spans across CI and CD.Developers write, deploy, and debug code faster with Cloud Code and Cloud ShellContinuously integrate and deliver updates with Cloud BuildContinuous delivery to GKE is made easier, faster, and more reliable with Cloud DeployDebug and troubleshoot with Google Cloud’s operations suiteYou can use your favorite partner solutions out of the boxMoreover, GKE Autopilot cluster accelerates app deployment reducing configuration time and simplifies ongoing management of Dev/Test clusters. You can read more on how to get started with GKE Autopilot. “Google Kubernetes Engine is easy to configure, and scales really well. That means the developers don’t need to think about managing it in production, they can simply set the parameters and be confident it will work.”—Vincent Oliveira, CTO, Lucky CartBolsters security into software supply chainSecurity remains a top of mind for all organizations. Kubernetes clusters created in the Autopilot mode implement many GKE hardening features by default. Furthermore, GKE Autopilot improves cluster security, restricting access to the Kubernetes API, prevents node mutation, enforcing robust security posture, and lets you implement additional guidance to harden security of your clusters. Binary Authorization is a deploy-time security control that ensures only trusted container images are deployed on GKE. With Binary Authorization, you can gain tighter control over your container environment by ensuring only verified images are integrated into the build-and-release process. You can read more on how to build security into your software supply chain”We needed to be HIPAA compliant, which was going to be painful on AWS, and we wanted to get away from managing and operating our own Kubernetes clusters,” recalled Astorino. “We had heard good things about GKE (Google Kubernetes Engine). And particularly valuable for us, — many technical requirements you need for HIPAA compliance are configured by default on Google Cloud.” —Troy Astorino, CoFounder & CTO of PicnicHealthCreates new opportunities with a platform approachModern application platforms spawn creativity and drive quick response to customer demands.  GKE customers use Kubernetes to build a modern enterprise-grade application platform for their organization. With the ability to achieve improved speed and performance for a variety of workloads through Tau VMs/GPU/TPU/Local SSD support, GKE helps them support a wide variety of containerized applications, including stateful and stateless, AI and ML, Linux and Windows. Only GKE can run 15,000 node clusters, outscaling other cloud providers by up to 10X, letting you run applications effectively and reliably at scale.”Google Cloud-managed services are playing a major role in enabling Noon.com customers to get their shopping done whenever they need it, without experiencing any delays or glitches, and without us having to lose sleep at night to ensure our platform is functioning as it should.”—Alex Nadalin, SVP of Engineering, Noon.comDelivers always-on experiences for customersConsumers today demand 24×7 digital experiences. GKE provides granular controls to deliver always-on, highly available, and reliable apps and services. With node auto-upgrade we automatically upgrade and patch your cluster nodes, while the control plane is always patched and upgraded by Google. You can also subscribe to a release channel – rapid, regular or stable – based on your needs and constraints. For enterprises, release channels provide the level of predictability needed for advanced planning, and the flexibility to orchestrate custom workflows automatically when a change is scheduled. You can learn more about release channels here, and about maintenance windows here.”To bring E.ON Optimum to market, we needed to transform in-house software into a highly scalable, reliable cloud-based solution. We were specifically looking for a cloud partner capable of running Kubernetes pods at scale and 100% of the time, and that led us to Google Cloud.”—Dennis Nobel, Digital Delivery Manager,E.ONEnables cost optimization and savings for organizationsIn the current macroeconomic environment, you often need to do more with fewer resources. GKE Autopilot dynamically adjusts compute resources, so there’s no need to figure out what size and shape nodes you should configure for your workloads. With GKE Autopilot, you pay only for the pods you use and you’re billed per second for vCPU, memory and disk resource requests. Moreover, GKE cost optimization insights help you discover optimization opportunities at scale, across your GKE clusters and workloads, automatically with minimal friction.”Since migrating to GKE, we’ve halved the costs of running our nodes, reduced our maintenance work, and gained the ability to scale up and down effortlessly and automatically according to demand. All our customer production loads and development environment run on GKE, and we’ve never faced a critical incident since.”—Helge Rennicke, Director of Software Development, Market Logic SoftwareFuels growth with focus on business innovationIT divisions are moving from cost centers to value centers by using managed cloud services. One can benefit from no-stress management and focus on business innovation using GKE Autopilot, which provides hands-off cluster management, SLA and eliminates most day-2 cluster operations. GKE delivers most dimensions of automation to efficiently and easily operate your applications. With fully managed GKE Autopilot,  combined with multi-dimensional auto-scaling capabilities, you can get started with a production ready secured cluster in minutes and have complete control over the configurations and maintenance.”The automated features of Google Kubernetes Engine enables us to manage app traffic and develop games at an amazingly high level of efficiency. Currently, we only need two engineers to monitor traffic volume and all the environments of our three games, which frees up more workforce for development and innovation work.”—Aries Wang, Research and Development Deputy Manager, Yile TechnologyGives freedom from proprietary tools for ITMulti-cloud is a reality. Proprietary tools often require specialized skills and lock you into huge licensing fees. You can minimize vendor lock-in and be well placed to maximize the benefits of a mutli-cloud strategy with conformant Kubernetes supported across multiple environments including all major cloud providers. Kubernetes’ workload portability provides you the flexibility to move your apps around without constraints.”MeilleursAgents is a product-oriented company and our goal is to deliver new services as fast as we can, in order to get market feedback, and improve them once they’re in production. Google Kubernetes Engine helps us do that by delivering flexibility and easy scaling, which is why we decided to make the switch.”—Thibault Lanternier, Head of Web Engineering, MeilleursAgentsJoin us at Building for the future with Kubernetes to kickstart or accelerate your Kubernetes journey. You’ll get access to technical demos that go deep into our Kubernetes services, developer tools, operations suite and security solutions. We look forward to partnering with you on your Kubernetes journey!Related ArticleWhy automation and scalability are the most important traits of your Kubernetes platformThe recipe for long-term success with Kubernetes: automation that matters and scalability that saves money.Read Article
Quelle: Google Cloud Platform

Why all retailers should consider Google Cloud Retail Search?

Cloud Retail Search, part of Discovery Solutions For Retail portfolio, helps retailers significantly improve the shopping experience on their digital platform with ‘Google-quality’ search. Cloud Retail Search offers advanced search capabilities such as better understanding user intent and self-learning ranking models that help retailers unlock the full potential of their online experience.Google Cloud’s Discovery Solutions For Retail are a set of services that can help retailers improve their digital engagement and are offered as part of our industry solutions.Executive SummaryRetailers are always working on trying to keep up with the ever changing consumer expectations and trying to forecast the next trend that can impact sales and revenue. The pandemic brought its own (and largely new) set of challenges which further complicated the issue over the last two years. The retailers were forced to adapt to the new consumer (low physical touch) behavior in which the browsing and product research was largely digital (endless aisle) and accelerated other trends such as buy online and pick up in stores (BOPIS), curbside pick up and pick up lockers. According to a McKinsey Global Survey from early last year, the pandemic has accelerated the pace of digital transformation by several years. The National Retail Federation (NRF) estimates that retail sales are expected to grow between 6% and 8% in 2022 (slower growth rate than in 2021), as consumers spend more on services instead of goods, deal with inflation and higher food & gas prices due to geopolitical disruptions in the world. And the competition continues to be fierce as ever. Amazon continues its dominance in the U.S. retail world and new PYMNTS data shows that Amazon’s share of US Ecommerce sales hit an all-time high of 56.7% in 2021.Customers now have more choices than ever on how they want to engage with the retailers, where they want to spend the money and make their purchase. They also have increased expectations from the retailers around providing a high quality product discovery experience, which is forcing the retailers to invest heavily on improving customer engagement on their digital platforms to boost conversion rate and overall customer loyalty.This is where Retail Search can help by providing an enhanced search experience that uses Google-quality search models to understand the customer intent and takes into account the retailer’s first party data (such as promotions, available inventory and price) for ranking results.How is Google Cloud Retail Search DifferentThe Ecommerce platform on-site search use case is not new and retailers have been trying to solve it effectively for the last two decades.  Most retailers recognize that search is a critical service on the platform and have spent countless resources to improve and fine tune it over the years. Yet the challenge remains. According to a Baymard Institute study in late as 2019, 61% of sites still required their users to search by the exact product type jargon the site uses. However, users now expect the same robust and intuitive search features as is offered by Google.com and other popular web platforms, who seem to have the uncanny ability to intelligently interpret and yield relevant results to complex search queries.Google’s decades of experience and research in search technology benefits Cloud Retail Search solution and that is what differentiates it from the competition. Advanced Query Understanding: Retail Search can provide more relevant results for the same query due to better query understanding features and knowing when to broaden or narrow the query results. While most search engines still rely largely on keyword based or matching tokens results, Retail Search has the advantage of being able to leverage Google search algorithms to return highly relevant results for product listings and category pages. Semantic Search: Intent recognition is a key requirement for semantic search and identifying what the customers mean when they enter the query is a key strength of Retail Search. This is critical for retailers since this has a direct impact on Clickthrough rate, Conversion rate and the Bounce Rate.Personalized Search Results: Another key differentiator for Retail Search is its ability to leverage user interaction data and ranking models to provide hyper personalized search results. Retailers are able to optimize search performance to deliver desired outcomes: better engagement, revenue, or conversions. Self-Learning and Self-Managed Solution: Retail Search models get better over time because of the self-learning capabilities built into the solution. In addition, the service is fully managed, which saves precious resources needed to keep it running and managing its set up.Strong Security Controls: The service runs on Google Cloud and follows security best practices to keep our customers’ data secure. Google never shares model weights or customer data across customers using the Retail API or other Discovery Solution products. For more details about this data use, see a description of Retail API data use.High Level Conceptual ViewHere is a simplified high-level view of Retail Search API. Retailers can call the API for the given search query and get back the results which can then be displayed on their digital properties.The returned results contains two types of information:Search results: Query search results including product listings and category pages based on advanced query understanding and semantic search.Dynamic faceted search attributes: Faceted Search is a feature that allows further refinement of the search by providing ways to apply additional filters while returning results. Retail Search needs the following datasets as input to train its machine learning models for search:Product Catalog:  Information about the available products including product categories, product description, in-stock availability, and pricing.User Events: This is the clickstream data that contains user interaction information such as clicks and purchases.Inventory / Pricing Updates: Incremental updates to in-stock availability and pricing as that information is updated.(Keeping the product catalog up to date and recording user events successfully is crucial for getting high-quality results. Set up Cloud Monitoring alerts to take prompt action in case any issues arise).Retailers also have the ability to set up business/config rules to customize their search results and optimize for business revenue goals such as Clickthrough rate, Conversion rate, Average size order etc.How to get startedRetail Search is generally available now and anyone with a Google cloud account can access it. If you don’t already have an account, you can start with a trial account for free here.Establish a Success Criteria: It’s important to establish a success criteria for measuring the effectiveness of Retail search. Get a consensus on which factor(s) you want to include in scope for measuring the effectiveness of Retail Search. This could include one or two from the following: Search Conversion Rate, Search Average Order Value, Search Revenue Per Visit and Null Search Rate (No Results Found).Initial Set Up: Create a Google Cloud Project and set up the Retail API. When you set up the Retail API for a new project, the Google Cloud Console displays the following three panels to help you configure your Retail API project:Catalog: displays product catalog and a link to import catalog.Event: displays user events and a link to import historical user events.Serving configs: contains details on serving configuration and a link to create a new serving configuration.Measuring Performance: Retail dashboards provide metrics to help you determine how incorporating the Retail API is affecting the results. You can view summary metrics for your project on the Analytics tab of the Monitoring & Analytics page in Cloud Console. Set up A/B Experiments: To measure the performance of Retail Search with another search solution, you can set up A/B tests using a third-party experiment platform such as Google Optimize.Summary:As retailers try to navigate through the post-pandemic world where supply chain failures and digital transformation acceleration are major focus areas, they now also have to keep a close eye on the recent geopolitical challenges resulting in rising inflation and costs.While we can all agree that in-store shopping will continue to be a major source of revenue, it is also important for retailers to tweak the in-store experience for the digital world. Trends such as buy online and pick up in stores (BOPIS), curbside pick up and pick up lockers are here to stay.Given all the above, consumer engagement and digital experience is more important now than ever before. The cost of search abandonment is way too high and has both short and longer term impact. Retail Search is a great solution to help reduce churn, improve conversion and retention. It provides Google-quality search models to help understand customer intent and the retailers have the ability to set up business/config rules to optimize search results for business revenue goals such as Clickthrough rate, Conversion rate and Average size order.Related ArticleShoppers know what they want. Does your site’s search understand?With Retail Search, retailers can now build Google-quality search into their sites and apps to drive sales and conversions.Read Article
Quelle: Google Cloud Platform

Announcing Virtual Machine Threat Detection now generally available to Cloud customers

Today we are delighted to announce that our unique, first-to-market detection capability with Virtual Machine Threat Detection (VMTD) in Security Command Center is now generally available for all Google Cloud customers. We launched this service six months ago in public preview and have seen a lot of enthusiasm from our customers. We’ve seen adoption from users around the world and in every industry. For years, we have said security must be engineered in, not bolted on. By baking this capability into our virtualization stack we are living up to our promise of delivering invisible security.  Our team has been busy scaling the service, refining our detection capabilities, and preparing our next major feature set. VMTD in general availability has been scaled to support significantly more frequent scanning across a tremendously large number of instances. Scaling the scanning of memory from the Google Cloud Compute Engine (GCE) fleet has posed unique challenges, and we’ve invested in caching scan results to enable more frequent scans of smaller – but more important – sections of memory. For customers, enabling VMTD is as easy as checking a box in their Security Command Center Premium settings. They consistently report that this is a game-changer compared to the challenges associated with third-party agent deployment. Because VMTD is deployed from the hypervisor, rather than inside the instance, our instrumentation is not as exposed to adversaries as are traditional endpoint detection and response technology (EDR) agents. This is an invisible-to-adversaries approach: they can not detect when we scan. By enabling VMTD, our customers have activated protection against millions of compute instances with a few simple checkboxes, and without using compute overhead.Deploying VMTD to Cloud customers has uncovered multiple attacks, and a few surprising false positives. In one example, our YARA rules detected a customer’s antivirus agent as mining cryptocurrency because that nameless agent happened to resolve a lot of mining pool domains as part of its protection scheme. Our YARA rules are developed in collaboration with the best of Google’s threat intelligence community, drawing on expertise from Google’s Threat Analysis Group as well as Google Cloud Threat Intelligence. As a cloud-native managed service, we’re always on the lookout for these cases and work hard to improve our service for all customers. With this release we are thrilled to announce that in addition to detecting mining on a particular instance, we now can–in the vast majority of instances–identify and report on the specific process that is engaged in mining activity. VMTD is now able to deeply understand the Linux kernel to report specific details about the threat execution we’ve observed without an agent inside the instance. This can enable easier triage, investigation, and response to our detection. In the academic literature of virtual machine introspection, understanding kernel data structure meaning from hardware contents is known as “bridging the semantic gap.” With this release, VMTD can bridge this gap and prove that insights useful to security teams can be pulled from the contents of memory for any given virtual machine.In the future, we plan on further improving VMTDs understanding of Linux kernels to detect additional advanced attacks and report live telemetry to our customers. With its unique position as an outside-the-instance observer, VMTD can detect rootkits and bootkits, attacks that tamper with kernel integrity and otherwise blind the kernel and EDR to their presence. We are delighted with the progress we’ve made on detecting the most common threats in Cloud computing environments. If you’re interested in trying out our service, navigate to the settings page of Security Command Center Premium and turn on Virtual Machine Threat Detection. You can read more about Security Command Center Premium and Virtual Machine Threat Detection at our docs pages.Related ArticleProtecting customers against cryptomining threats with VM Threat Detection in Security Command CenterExtending threat detection in Security Command Center with Virtual Machine Threat Detection.Read Article
Quelle: Google Cloud Platform