Cloud Computing Köln - Seite 2605 von 7200 - Neues zu Cloud Computing, Internet of Things und Technologien

With data volumes constantly growing, many companies find it difficult to use data effectively and gain insights from it. Often these organizations are burdened with cumbersome and difficult-to-maintain data architectures. One way that companies are addressing this challenge is with change streaming: the movement of data changes as they happen from a source (typically a database) to a destination. Powered by change data capture (CDC), change streaming has become a critical data architecture building block. We recently announced Datastream, a serverless change data capture and replication service. Datastream’s key capabilities include:Replicate and synchronize data across your organization with minimal latency. You can synchronize data across heterogeneous databases and applications reliably, with low latency, and with minimal impact to the performance of your source. Unlock the power of data streams for analytics, database replication, cloud migration, and event-driven architectures across hybrid environments.Scale up or down with a serverless architecture seamlessly. Get up and running fast with a serverless and easy-to-use service that scales seamlessly as your data volumes shift. Focus on deriving up-to-date insights from your data and responding to high-priority issues, instead of managing infrastructure, performance tuning, or resource provisioning.Integrate with the Google Cloud data integration suite. Connect data across your organization with Google Cloud data integration products. Datastream leverages Dataflow templates to load data into BigQuery, Cloud Spanner, and Cloud SQL; it also powers Cloud Data Fusion’s CDC Replicator connectors for easier-than-ever data pipelining.Click to enlargeDatasteam use casesDatastream captures change streams from Oracle, MySQL, and other sources for destinations such as Cloud Storage, Pub/Sub, BigQuery, Spanner and more. Some use cases of Datastream: For analytics use Datastream with a pre-built Dataflow template to create up-to-date replicated tables in BigQuery in a fully-managed way.For database replication use Datastream with pre-built Dataflow templates to continuously replicate and synchronize database data into Cloud SQL for PostgreSQL or Spanner to power low-downtime database migration or hybrid-cloud configuration.For building event-driven architectures use Datastream to ingest changes from multiple sources into object stores like Google Cloud Storage or, in the future, messaging services such as Pub/Sub or Kafka Streamline real-time data pipeline that continually streams data from legacy relational data stores (like Oracle and MySQL) using Datastream into MongoDB. How do you set up Datasteam?Create a source connection profile.Create a destination connection profile.Create a stream using the source and destination connection profiles, and define the objects to pull from the source.Validate and start the stream.Once started, a stream continuously streams data from the source to the destination. You can pause and then resume the stream. Connectivity optionsTo use Datastream to create a stream from the source database to the destination, you must establish connectivity to the source database. Datastream supports the IP allowlist, forward SSH tunnel, and VPC peering network connectivity methods.Private connectivity configurations enable Datastream to communicate with a data source over a private network (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a Virtual Private Cloud (VPC) peering connection. For a more in-depth look into Datastream check out the documentation.For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.
Quelle: Google Cloud Platform

31. Juli 2021

da Agency

Compliance Engineering – From manual attestation to continuous compliance

Risk Management and Compliance is as important in the cloud as it is in conventional on-premises environments. To help organizations in regulated industries meet their compliance requirements, Google Cloud offers automated capabilities that ensure the effectiveness of productionalization processes. Continuous compliance in the banking industryBanks have a formidable responsibility in managing the world’s wealth, and are therefore champions in diligently managing risk. Financial regulators in turn publish banking regulations to ensure banks assess and manage their risks accurately. Since banks are heavily reliant on information technology (IT), these regulations also cover the use of IT within banks.Regulated industries typically have an extended governance framework to ensure their deployed IT assets comply with the regulations, have a managed security posture and meet corporate risk appetites. Before a new application can be deployed in production, IT application owners typically look at a historical duration of several months to complete the necessary regulatory evidence. Control questions are typically based on the architectures of conventional on-premises technologies, and often lack relevance to cloud-specific technologies and hence do not benefit from using cloud automation capabilities. For example, current IT models within many banks are built to have only a few changes per month, whereas the cloud is capable of rolling out hundreds of changes every day. Let’s hear from one of the top regulated financial institutions what their challenges were before starting the transformation:“It was not just some of the significantly different technologies we’d be operating on and within, it was the foundational approach of having strong controls and control solutions embedded within the cloud platform. The changes in operating model from adopting Google Cloud made it evident to us that we’d need to revisit each and every control within our current control set.”—Bill Walker – Head of Operational Readiness at Deutsche BankThe following sections will help chief security and compliance officers assess their current estate and start the transformation of their IT-related risks with a set of key recommendations. Transforming processes from On-Premises to CloudThe objectives behind existing controls may still be relevant, however the definition and attestation often need to evolve to accurately address the operational risk. The strict control environment in combination with the ability for speed and go-to-market emphasizes the importance of effective controls and automated attestation in a cloud-based environment. For a broader digital transformation in regulated environments please refer to the Google Cloud whitepaper “Risk Governance of Digital Transformation in the Cloud”.Before we deep dive into the topic, let’s define some terms in this context. A control in its core helps to manage different types of risks. Security controls focus on addressing the risk of lapses of confidentiality, integrity and availability of information. Compliance controls focus on addressing the risk of failure to act in accordance with industry laws, regulations and internal policies. The fulfilment of a control is often reached by evidencing one or multiple underlying control questions. Group the controlsThe highly integrated services of the cloud allow the application owner to focus on the application relevant controls, while underlying platform services should be already evidenced centrally for the entire workload landscape. The following proposed grouping of controls will result in a reduction of controls every single workload has to evidence. Control owners and engineering teams can focus on the group of controls within their specialization, in other words the corresponding application engineers may not need to have full awareness of the implementation on the platform layer.The group of enterprise-wide controls are part of a vendor risk assessment assessing the cloud provider and cloud services. The evidence for these controls is not influenced by how the services would be configured or used within the corporation. A practical example is the provider’s employee on- and off-boarding process.The group of platform-wide controls are automatically enforced in each workload running on top of this landing zone. Practical examples are audit logging (on Org and Folder level), privileged user access management (PUAM) or encryption type used for data at rest. The use of Organization Policies allow the definition of configurations across the whole GCP resource hierarchy. The group of workload specific controls are evidenced on application level and focus on the custom application architecture. The evidenced configurations are specific to the deployed application and can include the used authentication providers, user access management and disaster recovery setup. In large landscapes an additional group of workload class would allow for clustering application specific controls by commonalities like processed data confidentiality or internet facing networks.Figure 1 – Grouping of controlsThe grouping helps to place the automation in the right stage of the productionalization. Enterprise wide controls are often only assessed once, so there is no big return on investing in automation. Platform wide controls should be automated in the Security Posture Management system allowing for continuous and close to real-time compliance auditing across all applications. Workload or Workload class specific controls should find their automation as part of the Continuous Integration / Continuous Deployment (CICD) pipeline. Assess cloud adequacyCloud offers many capabilities to make IT more secure, however many companies see their Cloud adoption as an opportunity to address technical debt and decrease risk posture in one go. Moving workloads to the Cloud influences the types of risks which need to be managed but should not lead to a less secure environment. Therefore it is key to review the existing controls and control questions to be effective.Let me illustrate this in a practical example where an existing on-prem control would check for a manual or automated procedure to deploy an application release including roll-back section for the failure case. Infrastructure as Code automates the end-to-end deployment and roll-back of an application, this means a Terraform script should be sufficient to evidence this control. Another example in the same area would be the governance around production access for engineers (i.e. approval process, managing lists who has access) will change significantly when human access to production infrastructure is by exception only.In short the controls and control questions have to be assessed to be:Effective – Control accurately evidences the cloud environmentAdjustment required – Control is relevant but has to be adapted to reflect cloud technologyObsolete – Control is not effective for assessing the cloud environment, can be deprecatedPublicly available cloud control frameworks form a solid base to incorporate into your specific controls and assessing for cloud adequacy. Furthermore possible gaps in your control framework can be identified and new adequate cloud native controls being introduced. Move more ops into devThe transformation of the productionalization process is not only about technology and compliance frameworks but also about people. People who have been maintaining and strengthening the current process for a long time. The control owners ultimately responsible for their control area may not be fully versed in the new Cloud technology, therefore they might naturally be a little sceptical. The transformation stands and falls with their involvement.Make the control owners get confident with the technologies by allocating education time, make them part of the design and engineering process from the very beginning and turn them into advocates for the cloud transformation in their respective organization. This recommendation follows the Site Reliability Engineering practice to make operations part of the development team. The benefit is that control owners can get confident in the technology and are sure their control is properly assessed. As the controls stem from different organizations (Security, Business Continuity, Regulations, etc.) they in turn will advocate for the control change in their respective organization.Have traceability of the controls and clear pass/fail criteriaClear traceability of the control questions to controls, to policies and to regulations help to clean up interpretations and enable large-scale automation.This one may sound obvious, however productionalization processes have grown over the years and sometimes it’s not entirely clear what risk shall be assessed and how the provided evidence is being used. In order to allow the application owners to move to production as quickly as possible it’s inevitable that controls are automated. Automation is only possible with clear pass/fail criteria without any aspect of interpretation. Controls which existed due to historical reasons without any clear traceability can be filtered and potentially missing controls can be identified.Banks excel at managing their risk as they have the great responsibility in managing the world’s wealth. With their digital transformation into the cloud, banks are facing the challenge to adapt their existing controls processes and automate the attestation. The control has to be understood, made effective for the new IT delivery paradigm, automated to accelerate the migration and moved to a continuous compliance model. Subsequent articles in this series will explore the concrete automation case studies and show how “infrastructure as code” and “compliance as code” allow regulatory audits to move from a cyclical assessment to continuous posture management during the entire lifetime of an IT asset.Related ArticleNew Paper: Assuring Compliance in the CloudToday we are releasing the new paper by the Office of the CISO of Google Cloud. In the paper we reveal a new approach for modernizing you…Read Article
Quelle: Google Cloud Platform

31. Juli 2021

da Agency

Introducing improved maintenance policy for Cloud Memorystore

Maintenance is a critical component of every database user experience as it ensures that your database is staying up to date with security patches, receiving feature updates, and improving performance. However, maintenance downtime can be impactful, especially when it occurs at inopportune times. We are happy to announce that Cloud Memorystore now enables you to have more control over when your Cloud Memorystore for Redis instances undergo routine maintenance. What is Cloud Memorystore Maintenance? Cloud Memorystore instances undergo periodic maintenance to keep your database performant and secure. These may include operating system patches, minor version upgrades, new features, and more. When this happens, your instance will experience disruptions. The nature of the disruption will vary depending on how you are using the service: Cloud Memorystore for Redis Standard Tier users will experience a failover event where clients will need to reconnect to the new primary instance Cloud Memorystore Redis Basic Tier users will experience a full cache flush What maintenance controls are available? To control the impact of maintenance updates, Cloud Memorystore is offering both maintenance rescheduling and advanced notification for critical maintenance updates. If you are already a Cloud SQL user, you are most likely already familiar with these controls. These features are currently available in Preview for all Cloud Memorystore for Redis users. For each Cloud Memorystore instance, you may set an optional preferred maintenance window when updates are scheduled. Once an update is ready, it will automatically be scheduled to take place during your preferred maintenance window. We recommend choosing a period where application traffic has been historically low. For example, a food ordering application might select an overnight window when their application is unused due to restaurants being closed for the night. In addition to selecting the preferred maintenance window, users can subscribe to maintenance notifications for advanced notice when an update is scheduled. After an update is scheduled, subscribed users will receive an email notification with the date and time of the scheduled maintenance. At this point, you can begin to plan for the upcoming maintenance update, opt to undergo maintenance sooner than the scheduled date, or defer maintenance by up to one week after the originally scheduled time. Getting started with Cloud Memorystore’s new maintenance policy Let’s start by setting a preferred maintenance window for your instance. This can be done during instance creation or by editing your existing instance. On the Cloud Memorystore Instance create or edit page, find the “Maintenance” section and click “Edit”. You can then specify a preferred day and start hour as shown here:Next, we recommend opting in to email notifications. You’ll start by navigating to the Cloud Console Communications Page. Select “ON” under Email for Cloud Memorystore. When notifications are enabled, you’ll get an email at least seven days before a scheduled update. To receive notifications, you must specify a preferred maintenance window.After an update is scheduled, the email notification will contain details on rescheduling the update if the scheduled time is not acceptable. This information is also visible in your cloud console via the instance details page as shown here:To learn more, you can find a detailed overview of Cloud Memorystore’s maintenance policy and how-to steps in our documentation. What’s next for Cloud Memorystore Improving our maintenance policy has been a highly requested feature for Cloud Memorystore. Cloud Memorystore recommends that all users opt-into the feature to ensure the smoothest possible maintenance experience. You can look forward to Cloud Memorystore for Memcached support in the near future as well as more advanced notification. Let us know what other features and capabilities you need with our Issue Tracker. We look forward to your feedback!Related ArticleIntroducing more maintenance controls for Cloud SQLThe Cloud SQL fully managed database service now lets you control routine maintenance tasks with advanced notification and maintenance re…Read Article
Quelle: Google Cloud Platform

31. Juli 2021

da Agency

Monitor models for training-serving skew with Vertex AI

This blog post focuses on how Vertex AI enables one of the core aspects of MLOps: monitoring models deployed in production for training-serving skew.Vertex AI, a managed platform that allows companies to accelerate the deployment and maintenance of artificial intelligence (AI) models.Here we will describe how Vertex AI makes it easy to:Turn on skew detection for a model deployed in Vertex AI’s Online Prediction service. No prior pre-processing tasks are required. Just run a command with a few basic parameters to turn on monitoring.Get alerted when data skew is detected.Visualize the skew in a console UI to quickly diagnose the issue and determine the appropriate corrective action.Model Monitoring explained in one minute (Cartoons by courtesy of Google Comics Factory)What is training-serving skew and how does it impact models deployed in productionHere is a definition of training-serving skew (from Rules of Machine Learning: Best Practices for ML Engineering):Training-serving skew is a difference between model performance during training and performance during serving. This skew can be caused by:A discrepancy between how you handle data in the training and serving pipelines.A change in the data between when you train and when you serve.A feedback loop between your model and your algorithm.In this blog post we will focus on tooling to help you identify the issues described by the first two bullets above: any change in the data (feature values) between training and serving, also known as data drift or covariate shift. The feedback loop problem mentioned in the third bullet point has to be addressed by proper ML system design. Please refer to this blog post for a description of how the Vertex Feature Store can help avoid this feedback loop problem.Changes in the input data can occur for multiple reasons: a bug inadvertently introduced to the production data pipeline, a fundamental change in the concept the model is trained to predict, a malicious attack on your service, and so on.Let’s look at a few real-world examples that impacted Google applications in the past. This paper — Data Validation for Machine Learning — describes the following incident:A ML pipeline trains a new ML model every dayAn engineer does some refactoring of the serving stack, inadvertently introducing a bug that pins a specific feature to -1Because the ML model is robust to data changes, it doesn’t output any error and continues to generate predictions, albeit with lower accuracyThe serving data is reused for training the next model. Hence the problem persists and gets worse until it is discovered.As this scenario illustrates, training-serving skew can sometimes be as harmful as a P0 bug in your program code. To detect such issues faster, Google introduced a rigorous practice of training-serving data skew detection for all production ML applications. As stated in this TFX paper:Let’s look at how this practice helped Google Play improve app install rate:By comparing the statistics of serving logs and training data on the same day, Google Play discovered a few features that were always missing from the logs, but always present in training. The results of an online A/B experiment showed that removing this skew improved the app install rate on the main landing page of the app store by 2%. (from TFX: A TensorFlow-Based Production-Scale Machine Learning Platform)Thus, one of the most important MLOps lessons Google has learned is: continuously monitor model input data for changes. For a production ML application, this is just as important as writing unit tests.Let’s take a look at how skew detection works in Vertex AI. How is skew identifiedVertex AI enables skew detection for numerical and categorical features. For each feature that is monitored, first the statistical distribution of the feature’s values in the training data is computed. Let’s call this the “baseline” distribution.The production (i.e. serving) feature inputs are logged and analyzed at a user determined time interval. This time interval is set to 24 hours by default, and can be set to any value greater than 1 hour. For each time window, the statistical distributions of each monitored feature’s values are computed and compared against the aforementioned training baseline. A statistical distance score is computed between the serving feature distribution and training baseline distribution. JS divergence is used for numerical features and L-infinity distance is used for categorical features. When this distance score exceeds a user configurable threshold, it is indicative of skew between the training and production feature values.Measuring how much the data changedSetup monitoring by running one simple commandOur goal is to make it very easy to turn on monitoring for a model deployed on Vertex AI’s Prediction service; almost as easy as just flipping a switch. Once a prediction endpoint is up and running, one can turn on training-serving skew detection by running a single gcloud command (and soon via a few clicks in the UI); no need for any pre-processing or extra setup tasks.To setup skew detection for a prediction endpoint, simply run a gcloud command such as:Let’s look at some of the key parameters (full gcloud docs are available here):emails: The email addresses to which you would like monitoring alerts to be sentendpoint: the prediction endpoint ID to be monitoredprediction-sampling-rate: For cost efficiency, it is usually sufficient to monitor a subset of the production inputs to a model. This parameter controls the fraction of the incoming prediction requests that are logged and analyzed for monitoring purposesdataset: For calculating the baseline, you can specify the training dataset via one of four options: a BigQuery table, a CSV file on Cloud Storage, a TFRecord file on Cloud Storage, or a managed dataset on Vertex AI. Please review the gcloud docs for information about the parameters “bigquery-uri”, “dataset”, “data-format” and “gcs-uris”. target-field: This specifies the field or column in the training dataset (also sometimes referred to as the ‘label’), that the model is trained to predict. monitoring-frequency: The time interval at which production (i.e. serving) inputs should be analyzed for skew. This is an optional parameter. It is set to 24 hours by default.feature-thresholds: Specify which input features to monitor, along with the alerting threshold for each feature. The alerting threshold is used to determine when an alert should be thrown. This is an optional parameter. By default, a threshold of 0.3 is used for each feature.Get alerts and visualize data in the console UIWhen a skew is detected for a feature, an alert is sent via email. (More ways of receiving alerts will be added in the near future, including mechanisms to trigger a model retraining pipeline). Upon getting an alert, users can log into the console UI to visualize and analyze the feature value distributions. Users can perform side by side visualization of the production data distributions and training data distributions, to diagnose the issue.Next StepsNow Model Monitoring is available as Preview. Anyone can try it from the Model Monitoring document, and there is also a great instruction demo video and example notebook created by Marc Cohen that provides the end-to-end scenario from deploying a model to an Endpoint to monitor the model with Model Monitoring. Take the first step into the real-world MLOps with Google’s best practices for productionizing ML systems.Related ArticleKickstart your organization’s ML application development flywheel with the Vertex Feature StoreA Feature Store is a key ingredient for MLOps, helping accelerate development, deployment, and management of production ML applications. …Read Article
Quelle: Google Cloud Platform