Kategorie: Google Cloud Platform

Cloud CISO Perspectives: June 2021

It’s been another busy month for security teams around the globe with no signs of slowing down. Many of us virtually attended RSA, and ransomware attacks continue to dominate headlines. The Biden Administration’s Executive Order on Cybersecurity is officially underway, with important milestones like the NIST workshops where many of us discussed the Standards and Guidelines to Enhance Software Supply Chain Security.In this month’s post I’ll recap these topics, the latest security updates from our Google Cloud product teams, and more. Don’t forget we have a new newsletter sign up for this series, so you can get the latest updates delivered to your inbox. Thoughts from around the industry Post-RSA takeaways: Resilience was the theme of this year’s event, and it’s something we need to think about throughout the rest of this year and beyond. Last year Iwrote about general resilience, and how one of the common mistakes many organizations make is to think that resilience can be obtained by simply writing down plans and procedures on what to do and how to respond to specific events. Overall, as recent cyber events have shown us, there are potential problems with this approach. For example, most crises or significant events are unique, and if the organization can only respond to what is in a plan or procedure then the muscle-memory for the needed agility in response may not be there. In many ways, what we need are a set of foundational capabilities across prevention, detection, response and recovery that are continuously exercised and improved. As everyone is learning, scenario specific plans are necessary but real resilience comes from organization muscle-memory joined with continuously tested people, process and technology capabilities that can be adapted to meet any challenge.Zero trust was also a hot topic during this year’s event. Between the COVID-19 pandemic and recent cyber attacks, it’s promising to see that organizations everywhere are now realizing they need a comprehensive and modern zero trust access approach that removes overreliance on the network perimeter to protect themselves against a variety of threats. For Google, zero trust is more than a marketing buzzword or trend to attach to—it’s how we have operated and helped to protect our internal operations over the last decade with our BeyondCorp framework. We will continue to improve upon the industry standard with our lessons learned, so that other organizations can benefit from zero trust access platforms with BeyondCorp Enterprise and move toward a safer security posture. Ransomware: From Colonial Pipeline to JBS, rarely a day goes by without a new attack in the news. The reality is that many of these problems stem from a lack of rigor implementing a range of basic technology controls. We’re at an inflection point where both the private and public sector need to work together to prioritize the right defenses against these rising threats. We think it’s a mistake to assume one control or one product can be the solution to ransomware. Many organizations have started to realize you need an array of controls working together to create and sustain a defensible security posture. We recently highlighted our recommendations to protect against ransomware based on the National Institute of Standards and Technology (NIST) primary pillars for a successful and comprehensive cybersecurity program. Securing open source software: The Open Source team at Google recently announced an incredibly useful exploratory visualization site called Open Source Insights, which provides an interactive view of the dependencies, including first layer and transitive dependencies, of open source projects. This is an extremely important effort for the industry, especially as more and more organizations rely on open source software for critical aspects of their environments. While the benefits of open source software are clear, challenges persist.  Take for example the complexity of the supply chain; open source software projects often have many hundreds of dependencies. Open Source Insights gives developers a comprehensive visualization of a project’s dependencies and their properties and vulnerabilities. This includes interactive visualizations for developers to analyze transitive dependency graphs, and a comparison tool to highlight how different versions of a package might affect their dependencies by introducing or removing licenses, fixing security problems, or changing the packages’ own dependencies. While much more work and research is needed in this space, Open Source Insights is a critical step in helping secure the open source software supply chain.Click to enlargeThe EU Cloud Code of Conduct: While it went into force in 2018, the EU’s General Data Protection Regulation (GDPR) remains firmly top of mind as organizations use the cloud for processing of sensitive data. Providers like Google Cloud are often asked derivatives of the question “how can we be sure you’re taking appropriate measures to safeguard data under the GDPR.” We now have a definitive answer. The EU GDPR Cloud Code of Conduct(CoC) is a mechanism for cloud providers to demonstrate how they offer sufficient guarantees to implement appropriate technical and organizational measures as data processors under the GDPR. The Belgian Data Protection Authority, based on a positive opinion by the European Data Protection Board (EDPB), last month approved the CoC, a product of years of constructive collaboration between the cloud computing community, the European Commission, and European data protection authorities. This is the first European code approved under the GDPR; it is excellent news for the industry to have a new transparency and accountability tool that helps promote trust in the cloud. We are proud to say that Google Cloud Platform and  Google Workspace already adhere to these provisions.Spotlight on the Administration’s Executive Order on Cybersecurity The Presidential Administration’s recent moves in the Executive Order to shore up our nation’s cyber defenses is an important milestone for both public and private sector organizations. At Google, we are deeply committed to advancing cybersecurity issues and believe that government officials shouldn’t have to tackle these issues on their own. Importantly, the EO makes critical strides to help modernize government technology and advance security innovation as well as improve standards for secure software development. We’ve already shared our perspective with the government and will continue to advocate on these issues in the coming months. Modernization and security innovation: One of the most promising aspects of the government’s approach is to set agencies and departments on a path to modernize security practices and strengthen cyber defenses across the federal government. For too long, the public sector has tried to solve security challenges by spending more on security products, but as recent events have proved, spending billions of dollars on cybersecurity on an unmodernized IT platform is like building on sand. We strongly support this push towards modernization and agree with the government’s focus on making security simple and scalable, by default. Modernizing can not only build cybersecurity at a foundational level but also gives the federal government the opportunity to diversify their vendors, which can lead to improved resilience.Secure software development: Earlier this month Google participated in the NIST workshops and submitted position papers for how the industry can enhance software supply chain security. We believe that the government’s call to action on secure software development practices could bring about the most significant progress on cybersecurity in a decade and will likely have the biggest impact on the government’s risk posture in the long term. To further the adoption of supply chain integrity best practices, Google in collaboration with the OpenSSF has proposed Supply-chain Levels for Software Artifacts (SLSA) to formalize criteria around software supply chain integrity. We look forward to continuing to collaborate and engage with the Administration on this important work.Google Cloud Security highlights Google Cloud Named a Leader in Forrester Wave™: Unstructured Data Security Platforms: Providing effective controls to protect sensitive data in the cloud is a core part of our Google Cloud product strategy and unstructured data in particular can be challenging to secure. Given the importance of these capabilities to our customers, we were happy to see that Forrester Research named Google Cloud a Leader in The Forrester Wave™: Unstructured Data Security Platforms, Q2 2021 report. The report evaluated the 11 most significant providers with platform solutions to secure and protect unstructured data, spanning cloud providers to data security-focused vendors. Google Cloud rated highest in the current offering category among all the providers evaluated and received the highest possible score in sixteen criteria. A copy of the full report can be viewed here.Security benefits of a Data Cloud: Last month, we held our first Data Cloud Summit where we announced three new services as part of our database and data analytics portfolio to provide organizations with a unified data platform: Dataplex, Analytics HubandDatastream. Security professionals often default to using only security branded tools, but some of the best tools for security teams might be to use data and analytics products that are key to other business functions within the organization. Digital technologies like AI, ML and data can be used to power innovation, especially for security efforts. At Google, security is the cornerstone of our product strategy and our customers can take advantage of the same secure-by-design infrastructure, built-in data protection, and global network that we use to ensure compliance, redundancy and reliability.New features to secure your Cloud Run environments:  Cloud Run makes developing and deploying containerized applications easier for developers. We announced several new ways to help make Cloud Run environments more secure based on enhanced integrations with Secret Manager, Binary Authorization, Cloud KMS, and Recommendation Hub.Advanced counter-abuse and threat analysis features in Google Workspace:We continue to add controls and capabilities for Workspace admins to protect their users and organizations against threats and abuse. We recently added features that enrich security alerts with VirusTotal threat context and reputation data, enable blocking of abusive users and bulk removal content they’ve shared in Drive, and programmatic blocking third-party API access.That wraps up another month of thoughts and highlights. If you’d like to have this Cloud CISO Perspectives post delivered every month to your inbox, click here to sign-up.Next month, we’ll be busy hosting our first digital Security Summit where you can hear from industry leaders and engage in interactive sessions that can help you solve your most critical security challenges. Be sure to register and tune in to the great event we have planned!Related ArticleCloud CISO Perspectives: May 2021Google Cloud CISO Phil Venables shares his perspective on industry news as RSA 2021 approaches.Read Article
Quelle: Google Cloud Platform

Creating custom financial indices with Dataflow and Apache Beam

Financial institutions across the globe rely on real-time indices to inform real-time portfolio valuations, to provide benchmarks for other investments, and as a basis for passive investment instruments including exchange-traded products (ETPs). This reliance is growing—the index industry dramatically expanded in 2020, reaching revenues of $4.08 billion.Today, indices are calculated and distributed by index providers with proximity and access to underlying asset data, and with differentiating real-time data processing capabilities. These providers offer subscriptions to real-time feeds of index prices and publish the constituents, calculation methodology, and update frequency for each index.But as new assets, markets, and data sources have proliferated, financial institutions have developed new requirements. Financial institutions will need to quickly create bespoke and frequently updating indices that represent a specific actual or theoretical portfolio, with its unique constituents and weightings.In other words, existing index providers and other financial institutions alike will need mechanisms for rapid creation of real-time indices. This blog post’s focus—an index publication pipeline collaboratively developed by CME Group and Google Cloud—is an example of such a mechanism. The pipeline closely approximates a particular CME Group index benchmark, but with far greater frequency (in near real time vs. daily) than its official counterpart. It does so by leveraging open-source models such as Apache Beam and cloud-based technologies such as Dataflow, which automatically scales pipelines based on inbound data volume.Machine learning’s production problemIn the past decade, advances in AI toolchains have enabled faster ML model training—and yet a majority of ML models are still not making it into production. As organizations endeavor to develop their ML capabilities, they soon realize that a real-world ML system is comprised of a small amount of ML code embedded in a network of complex and large ancillary components. Each component brings its own development and operational challenges, which are met by bringing a DevOps methodology to the ML system, commonly referred to as MLOps (Machine Learning Operations). To apply ML to business problems, a firm must develop continuous delivery and automation pipelines for ML.This index publication collaboration is instructive because it demonstrates MLOps best practices for just such a pipeline. One Apache Beam pipeline, suited for operating on both batch and streaming data, extracts insights and packages them for downstream consumers. These consumers may include ML pipelines that, thanks to Apache Beam, require only one code path for inference across batch and real-time data sources. The pipeline is run inside Google Cloud’s Dataflow execution engine, greatly simplifying management of underlying compute resources. But the collaboration’s value is not constrained to the ML and data science realm. The project shows that consumers of the Apache Beam pipeline’s insights may also include traditional business intelligence dashboards and reporting tools. It also demonstrates the simplicity and economy of cloud-based time series data such as CME Smart Stream, which is metered by the hour, quickly and automatically provisioned, and consumable at a per-product-code (not per-feed) level.A focus on real-time processing for financial servicesTo illustrate the above points, the collaboration applies data engineering and MLOps best practices to a financial services problem. We chose the financial services domain because many financial institutions do not yet have real-time market data processing or MLOps capabilities today, owing to a significant gap on either side of their ML/AI objectives.Upstream from ML/AI models, financial institutions often experience a data engineering gap. For many financial institutions, batch processes have sufficiently addressed business requirements. As a result, the temporal nature of the time series data underlying these processes is deemphasized. For example, the original purpose of most trade booking systems was to capture a trade and ensure that it found its way to the middle and back office for settlement. It was not built with ML/AI in mind, and its underlying data therefore has not been packaged for consumption by ML/AI processes. And downstream from ML/AI models, financial institutions often encounter the aforementioned “ML production problem.”As ML/AI becomes ever more strategic, these two gaps have left many financial institutions in a conundrum—unable to train ML models for lack of properly packaged time series data, and unmotivated to package time series data for lack of ML models. By recreating a key energy market index using open-source libraries and cloud-based tools, this collaboration demonstrates that for the financial services domain a solution to this conundrum is more accessible today than ever. Creating a new indexWe modeled our new index after one of CME Group’s many index benchmarks. The particular index expresses the value of a basket of three New York Mercantile Exchange—listed energy futures as a single price. Today, CME Group publishes the index at the end of the day by calculating the settlement price of each underlying futures contract, and then weighing and summing these values. While CME Group does not currently publish this index in real time, this collaboration aims to create a near real-time solution leveraging Google Cloud capabilities and CME Group market data delivered via CME Smart Stream. However, in order to publish the value so frequently—every five seconds, with 40-second publish latency—this collaboration’s pipeline has to solve a number of challenges in near-real time.First, the pipeline must process sparse data from three separate trades feeds in memory to create open-high-low-close (OHLC) bars. More specifically, for five-second windows for each of the three front-month (and sometimes second-month) energy contracts, a bar must be produced. This is solved by using the Apache Beam library to implement functions which, when executed on Dataflow, automatically scale out as input load increases. The bars must be time-aligned across the underlying feeds, which is greatly simplified by Beam’s watermark feature. And for intervals in which no tick data is observed, the Beam library is used to pull forward the last value received, yielding perfect gap-free bars for downstream processors.Second, the pipeline must calculate volume-weighted average price (VWAP) in near real-time for each front-month contract. The VWAP calculations are also written using the Beam API and executed on Dataflow. Each of these functions requires visibility of each element in the time window, so the functions cannot be arbitrarily scaled out. Nonetheless, this is tractable because their input—OHLC bars—is manageably small.Third, the pipeline must replicate CME Group’s specific settlement price methodology for each contract. The rules specify whether to use VWAP or another source as price, depending on certain conditions. They also specify how to weigh combinations of monthly contracts during a roll period. The pipeline again encapsulates these requirements as an Apache Beam class, and joins the separate price streams at the correct time boundary.The end result is a new stream publishing bespoke index data to a Google Cloud Pub/Sub topic thousands of times daily, enabling AI models as well as traditional industry index usage, dashboards, and other tools to assist real-time decision making. The stream’s pipeline uses open source libraries that solve common time series problems out-of-the box, and cloud-based services to reduce the user’s operational and scaling burden.Click to enlargeThe importance of cloud-based dataThe promise of cloud-based pipeline execution services cannot be realized using legacy data access patterns, which often require market data users to colocate and configure servers and network gear. Such patterns inject expense and scaling complexity into the pipeline’s overall operation, diverting resources from the adoption of MLOps best practices. Instead, a newer, cloud-based access pattern—in which resources subscribe to data streams inexpensively, rapidly and programatically—is necessary.In 2018, CME Group identified the customer need for accessible futures and options market data. CME Group collaborated with Google Cloud to launch CME Smart Stream, which distributes CME Group’s real-time market data across Google Cloud’s global infrastructure with sub-second latency. Any customer with a CME Group data usage license and a Google Cloud project can consume this data for an hourly usage fee, without purchasing and configuring servers and network gear.CME Smart Stream met this index pipeline’s requirements for cost-effective, cloud-based streaming data, but this is just one use case. Since the launch of a CME Smart Stream offering on Google Cloud, globally dispersed firms have adopted the solution. For example, Coin Metrics has been using the offering to better inform its customers in the crypto markets. According to CME Group, Smart Stream has become popular with new customers as the fastest, simplest way to access CME Group’s market data from anywhere in the world. Adapt the design pattern to your needsBy combining cloud-based data, open-source libraries, and cloud-based pipeline execution services, we created a real-time index using the same constituents as its end-of-day counterpart. Additionally, financial institutions will find this approach addresses many other challenges—real-time valuation of a large set of portfolios; benchmark creation for new ETPs; or external publication of new indices.Give it a tryThis approach is available to help you meet your organization’s needs. We’ll be discussing this topic in CME Group’s webinar End-to-End Market Data Solutions in the Cloud at 10:30 am ET on June 16th.
Quelle: Google Cloud Platform

Cloud Run: A story of serverless containers

Mindful Containers is a fictitious company that is creating containerized microservice applications. They need a fully managed compute environment for deploying and scaling serverless containerized microservices. So, they are considering Cloud Run. They are excited about Cloud Run because it abstracts away the cluster configuration, monitoring, and management so they can focus on building the features for their apps. Cloud Run is a fully-managed compute environment for deploying and scaling serverless containerized microservices.Click to enlargeWhat is Cloud Run?Cloud Run is a fully-managed compute environment for deploying and scaling serverless HTTP containers without worrying about provisioning machines, configuring clusters, or autoscaling.No vendor lock-in – Because Cloud Run takes standard OCI containers and implements the standard Knative Serving API, you can easily port over your applications to on-premises or any other cloud environment. Fast autoscaling – Microservices deployed in Cloud Run scale automatically based on the number of incoming requests, without you having to configure or manage a full-fledged Kubernetes cluster. Cloud Run scales to zero— that is, uses no resources—if there are no requests.Split traffic – Cloud Run enables you to split traffic between multiple revisions, so you can perform gradual rollouts such as canary deployments or blue/green deployments.Custom domains – You can set up custom domain mapping in Cloud Run and it will provision a TLS certificate for your domain. Automatic redundancy – Cloud Run offers automatic redundancy so you don’t have to worry about creating multiple instances for high availabilityHow to use Cloud RunWith Cloud Run, you write your code in your favorite language and/or use a binary library of your choice. Then push it to Cloud Build to create a container build. With a single command—“gcloud run deploy”—you go from a container image to a fully managed web application that runs on a domain with a TLS certificate and auto-scales with requests.How does Cloud Run work?Cloud Run service can be invoked in the following ways:HTTPS: You can sendHTTPS requests to trigger a Cloud Run-hosted service. Note that all Cloud Run services have a stable HTTPS URL. Some use cases include: Custom RESTful web APIPrivate microserviceHTTP middleware or reverse proxy for your web applicationsPrepackaged web applicationgRPC: You can usegRPC to connect Cloud Runservices with other services—for example, to provide simple, high-performance communication between internal microservices. gRPC is a good option when you: Want to communicate between internal microservicesSupport high data loads (gRPC uses protocol buffers, which are up to seven times faster than REST calls)Need only a simple service definition you don’t want to write a full client libraryUse streaming gRPCs in your gRPC server to build more responsive applications and APIsWebSockets: WebSockets applications are supported on Cloud Run with no additional configuration required. Potential use cases include any application that requires a streaming service, such as a chat application.Trigger from Pub/Sub:You can use Pub/Sub to push messages to the endpoint of your Cloud Run service, where the messages are subsequently delivered to containers as HTTP requests. Possible use cases include:Transforming data after receiving an event upon a file upload to a Cloud Storage bucketProcessing your Google Cloud operations suite logs with Cloud Run by exporting them to Pub/SubPublishing and processing your own custom events from your Cloud Run servicesRunning services on a schedule: You can use Cloud Scheduler to securely trigger a Cloud Run service on a schedule. This is similar to using cron jobs. Possible use cases include:Performing backups on a regular basisPerforming recurrent administration tasks, such as regenerating a sitemap or deleting old data, content, configurations, synchronizations, or revisionsGenerating bills or other documentsExecuting asynchronous tasks: You can use Cloud Tasks to securely enqueue a task to be asynchronously processed by a Cloud Run service. Typical use cases include:Handling requests through unexpected production incidentsSmoothing traffic spikes by delaying work that is not user-facingReducing user response time by delegating slow background operations, such as database updates or batch processing, to be handled by another service, Limiting the call rate to backend services like databases and third-party APIsEvents from Eventrac: You can trigger Cloud Run with events from more than 60 Google Cloud sources. For example:Use a Cloud Storage event (via Cloud Audit Logs) to trigger a data processing pipeline Use a BigQuery event (via Cloud Audit Logs) to initiate downstream processing in Cloud Run each time a job is completedHow is Cloud Run different from Cloud Functions?Cloud Run and Cloud Functions are both fully managed services that run on Google Cloud’s serverless infrastructure, auto-scale, and handle HTTP requests or events. They do, however, have some important differences:Cloud Functions lets you deploy snippets of code (functions) written in a limited set of programming languages, while Cloud Run lets you deploy container images using the programming language of your choice. Cloud Run also supports the use ofany tool or system library from your application; Cloud Functions does not let you use custom executables. Cloud Run offers a longer request timeout duration of up to 60 minutes, while with Cloud Functions the requests timeout can be set as high as 9 mins. Cloud Functions only sends one request at a time to each function instance, while by default Cloud Run is configured to send multiple concurrent requests on each container instance. This is helpful to improve latency and reduce costs if you’re expecting large volumes. PricingCloud Run comes with a generous free tier and is pay per use, which means you only pay while a request is being handled on your container instance. If it is idle with no traffic, then you don’t pay anything.ConclusionAfter learning about the ease of set up, scalability, and management capabilities of Cloud Run the Mindful Containers team is using it to deploy stateless microservices. If you are interested in learning more, check out the documentation.For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.devRelated Article3 cool Cloud Run features that developers loveCloud Run developers enjoy pay-per-use pricing, multiple concurrency and secure event processing.Read Article
Quelle: Google Cloud Platform

Ubuntu Pro lands on Google Cloud

Today, we’re pleased to announce the general availability of Ubuntu Pro images on Google Cloud, providing customers with an improved Ubuntu experience, expanded security coverage, and integration with critical Google Cloud features. In partnership with Canonical, we’re making it even easier for customers that have fully embraced open source to ensure security and compliance for their most mission-critical and enterprise workloads.With Ubuntu Pro on Google Cloud, you  now have access to features like:10-year lifetime security updates – Canonical backs Ubuntu Pro for 10 years with security updates and a guaranteed upgrade path.FIPS & CC-EAL2 certification – Ubuntu Pro includes components that meet requirements from entities like FedRAMP, HIPAA, ISO, and PCI.Open-source security coverage – Protect your most important open-source workloads including MongoDB, Apache Kafka, Redis, NGINX, and PostgreSQL.Multi-version availability – Pro images are available for the three most popular Ubuntu Server distributions: 16.04 LTS, 18.04 LTS, and 20.04 LTS.Kernel Livepatch – Kernel patches are delivered immediately without having to reboot your VMs.Optional CIS and DISA STIG profiles – Choose from two leading profiles to harden your environment according to industry benchmarks.Cloud-based pricing – Ubuntu Pro does not require a contract, and pricing tracks with the underlying compute cost depending on the instance type.Extended Security Maintenance (ESM) for Ubuntu 16.04 LTS with Ubuntu ProAvailability of Ubuntu Pro images is especially important if you’re an Ubuntu 16.04 LTS customer and want extended security maintenance (ESM) for your virtual machines but don’t want to upgrade to Ubuntu 18.04 LTS or Ubuntu 20.04 LTS versions immediately. ESM is included with Ubuntu Pro 16.04. You can move your workloads from Ubuntu 16.04 LTS VM instances to Ubuntu Pro 16.04 instances to continue receiving ESM and all the above-mentioned benefits, without having to test your applications on a new version of the OS.Gojek has evolved from offering just ride-hailing to a suite of more than 20 services today, serving everyday solutions for millions of users across Southeast Asia.“We needed more time to comprehensively test and migrate our Ubuntu 16.04 LTS workloads to Ubuntu 20.04 LTS, which would mean stretching beyond the standard maintenance timelines for Ubuntu 16.04 LTS. With Ubuntu Pro on Google Cloud, we now have the ability to postpone this, and in moving our 16.04 workloads to Ubuntu Pro, we benefit from its live kernel patching and improved security coverage for our key open source components.”—Kartik Gupta, Engineering Manager for CI/CD & FinOps at Gojek“With the launch of Ubuntu Pro on Google Cloud, we build on our joint investments with Google to optimize Ubuntu performance on Google Cloud, and add comprehensive security patching and Long Term Support for another 30,000 open source packages—the widest range of security-maintained open source on the planet,” said Mark Shuttleworth, CEO of Canonical. “As the world moves to open source for everything, Canonical offers the safety net of security maintenance that enterprises count on to unleash their developers.”Getting startedGetting started with Ubuntu Pro on Google Cloud is simple. You can now purchase these premium images directly from Google Cloud by selecting Ubuntu Pro as the operating system straight from the Google Cloud Console.To learn more about Ubuntu Pro on Google Cloud, please visit the documentation page and read the announcement from Canonical.
Quelle: Google Cloud Platform

Monitoring BigQuery reservations and slot utilization with INFORMATION_SCHEMA

BigQuery Reservations help manage your BigQuery workloads. With flat-rate pricing, you can purchase BigQuery slot commitments in 100-slot increments in either flex, monthly, or yearly plans instead of paying for queries on demand. You can then create/manage buckets of slots called reservations and assign projects, folders, or organizations to use the slots in these reservations. By default, queries running in a reservation automatically use idle slots from other reservations. In this way, organizations have greater control over workload management in a way that ensures high-priority jobs always have access to the resources they need without contention. Currently, two ways to monitor these reservations and slots are via the BigQuery Reservations UI or Cloud Monitoring.But how does an organization know how many slots to delegate to a reservation? Or if a reservation is being over or underutilized? Or what the overall slot utilization is across all reservations? In this blog post, we will discuss how we used BigQuery’s INFORMATION_SCHEMA system tables to create the System Tables Reports Dashboard and answer these questions.Using INFORMATION_SCHEMA tablesThe INFORMATION_SCHEMA metadata tables contain relevant, granular information about jobs, reservations, capacity commitments, and assignments. Using the data from these tables, users can create custom dashboards to report on the metrics they are interested in in ways that inform their decision making.While there are several tables that make up INFORMATION_SCHEMA, there are a few that are specifically relevant to monitoring slot utilization across jobs and reservations. The JOBS_BY_ORGANIZATION table is the primary table to extract job-level data across all projects in the organization. This information can be supplemented with data from the CAPACITY_COMMITMENT_CHANGES_BY_PROJECT, RESERVATION_CHANGES_BY_PROJECT, and ASSIGNMENT_CHANGES_BY_PROJECT tables to include details about specific capacity commitments, reservations, and assignments. It’s worth noting that the data retention period for INFORMATION_SCHEMA is 180 days and all timestamps are in UTC. For information about the permissions required to query these tables, follow the links above.Monitoring with the System Tables Reports DashboardThe System Tables Reports Dashboard is a Data Studio dashboard that queries data from INFORMATION_SCHEMA by using Data Studio’s BigQuery connector. Organizations can use this dashboard and/or its underlying queries as-is or as a starting point for more complex solutions in Data Studio or any other dashboarding tools.Daily Utilization ReportThe Daily Utilization Report gives an overview of an organization’s daily slot utilization measured in slot days. The primary chart in the report is for overall slot utilization per day alongside the active capacity commitments for the organization. This chart is ideal for gaining a high-level understanding of how an organization’s usage compares to the total number of slots it has committed to (or purchased).Click to enlargeThe query used to derive the average slot utilization is as follows:Slot utilization is derived by dividing the total number of slot-milliseconds (total_slot_ms from INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION) consumed by all jobs on a given day by the number of milliseconds in a day (1000 * 60 * 60 * 24). This aggregate-level computation provides the most accurate approximation of the overall slot utilization for a given day. Note that this calculation is most accurate for organizations with consistent daily slot usage. If an organization does not have consistent slot usage this number might be lower than expected. For more information about calculating average slot utilization, see our public documentation.This report also includes charts that break down the utilization further by job type, project id, reservation id (shown below), user email, and top usage.click to enlargeHourly Utilization ReportThe Hourly Utilization Report is similar to the daily utilization report but gives an overview of an organization’s hourly slot utilization measured in slot hours. This report can help an organization understand their workloads at a more granular level in a way that helps with workload management.Reservation Utilization ReportThe Reservation Utilization Report gives an overview of an organization’s current assignments and reservation utilization in the last 7 and 30 days.The current reservation assignments table displays details for the current assignments across an organization including the assignment type, job type and reservation capacity.Click to enlargeThe reservation utilization tables display information about the utilization of a given reservation in the last 7 or 30 days. This includes average weekly or monthly slot utilization, average reservation capacity, current reservation capacity, and average reservation utilization. Average weekly and monthly utilization is derived using the same calculation as daily utilization, but adjusted for a week or month accordingly. click to enlargeThese tables are great for understanding if an organization is making the most of its allocated reservations. Reservations that are severely over or under utilized are colored in red, while reservations that are close to 100% utilization are colored in green. That said, because idle slot capacity is shared across reservations by default, underutilized reservations do not necessarily indicate that slots are being wasted. Instead, the jobs in that reservation simply do not need as many slots reserved and those slots could be allocated to a different reservation.Job Execution ReportThe Job Execution Report provides a per-job breakdown of slot utilization, among other job statistics. The purpose of this report is to allow users to drill down into individual jobs or understand trends in a specific group of jobs.Click to enlargeIn this report, the average slot utilization is displayed on a per-job level instead of an aggregate level. This is calculated by dividing total_slot_ms for that job by the  job’s duration in milliseconds (this can be computed by subtracting creation_time from end_time) as seen in the following query.Job Error ReportThe Job Error Report provides an overview of the types of errors encountered by jobs in the organization aggregated by project and error reason, among other fields. The INFORMATION_SCHEMA tables provide detailed information about job-level errors, so depending on an organization’s use case this report can be customized with more specific error reporting information.Click to enlargeWhat’s next?To learn more about INFORMATION_SCHEMA and the System Tables Reports Dashboard, check out the videos in our Modernizing Data Lakes and Data Warehouses with GCP course on Coursera. For more detailed information about each report, the queries used, and how to copy the dashboard for your own organization, visit our Github Repository.
Quelle: Google Cloud Platform

Why you need to explain machine learning models

Many companies today are actively using AI or have plans to incorporate it into their future strategies — 76% of enterprises are now prioritizing AI and ML over other initiatives in their IT budgets and the global AI industry is expected to reach over $260 billion by 2027.  But as AI and advanced analytics become more pervasive, the need for more transparency around how AI technologies work will be paramount. In this post, we’ll explore why explainable AI (XAI) is essential to widespread AI adoption, common XAI methods, and how Google Cloud can help. Why you need to explain ML modelsAI technology suffers from what we call a black box problem. In other words, you might know the question or the data (the input), but you have no visibility into the steps or processes that provide you with the final answer (the output). This is especially problematic in deep learning and artificial neural network approaches, which contain many hidden layers of nodes that “learn” through pattern recognition. Stakeholders are often reluctant to trust ML projects because they don’t understand what they do. It’s hard for decision-makers to relinquish control to a mysterious machine learning model, especially when it’s responsible for critical decisions. AI systems are making predictions that have a profound impact, and in some industries like healthcare or driverless cars, it can mean the difference between life and death. It’s often hard to get support that a model can be trusted to make decisions, let alone make them better than a human can—especially when there is no explanation of how that decision was made. How did the AI model predict or make a decision? How can you be sure there is no bias creeping into algorithms? Is there enough transparency and interpretability to trust the model’s decision? Decision-makers want to know the reasons behind an AI-based decision, so they have the confidence that it is the right one. In fact, according to a PwC survey, the majority of CEOs (82%) believe that AI-based decisions must be explainable to be trusted.What is Explainable AI? Explainable AI(XAI) is a set of tools and frameworks that can be used to help you understand how your machine learning models make decisions. This shouldn’t be confused with showing a complete step-by-step deconstruction of an AI model, which can be close to impossible if you’re attempting to trace the millions of parameters used in deep learning algorithms. Rather, XAI aims to provide insights into how models work, so human experts are able to understand the logic that goes into making a decision. When you apply XAI successfully, it offers three important benefits: 1. Increases trust in ML modelsWhen decision-makers and other stakeholders have more visibility into how a ML model found its final output, they are more likely to trust AI-based systems. Explainable AI tools can be used to provide clear and understandable explanations of the reasoning that led to the model’s output. Say you are using a deep learning model to analyze medical images like X-rays, you can use explainable AI to produce saliency maps (i.e. heatmaps) that highlight the pixels that were used to get the diagnosis. For instance, a ML model that classifies a fracture would also highlight the pixels used to determine that the patient is suffering from a fracture.2. Improves overall troubleshooting Explainability in AI can also enable you to debug a model and troubleshoot how well a model is working. Let’s imagine your model is supposed to be able to identify animals in images. Over time, you notice that the model keeps classifying images of dogs playing in snow as foxes. Explainable AI tools make it easier to figure out why this error keeps occurring. As you look into your explainable AI models that you’re using to show how a prediction is made, you discover that that ML model is using the background of an image to differentiate between dogs and foxes. The model has mistakenly learned that domestic backgrounds are dogs and snow in an image means the image contains a fox. 3. Busts biases and other potential AI potholesXAI is also useful for identifying sources of bias. For example, you might have a model to identify when cars are making illegal left hand turns. When you are asked to define what the violation is based on in an image, you find out that the model has picked up a bias from the training data. Instead of focusing on cars turning left illegally, it’s looking to see if there is a pothole. This influence could be caused by a skewed dataset that contained a large amount of images taken on poorly maintained roads, or even real-bias, where a ticket might be more likely to be given out in an underfunded area of a city. Where does explainability fit into the ML lifecycle? Explainable AI should not be an afterthought that’s done at the end of your ML workflow. Instead, explainability should be integrated and applied every step of the way—from data collection, processing to model training, evaluation, and model serving. There are a few ways you can work explainability into your ML lifecycle. This could mean using explainable AI to identify data set imbalances, ensure model behavior satisfies specific rules and fairness metrics, or show model behavior both locally and globally. For instance, if a model was trained using synthetic data, you need to ensure it behaves the same when it uses real data. Or, as we discussed above with deep learning models for medical imaging, a common form of explainability is to create heatmaps to identify the pixels used for image classification.Another tool you might use is sliced evaluations of machine learning model performance. According to our AI principles, you should avoid creating or reinforcing unfair bias. AI algorithms and datasets can often reflect or reinforce unfair biases. If you notice that a model is not performing well for a small minority of cases, it’s important for you to address any fairness concerns. Sliced evaluations will allow you to explore how different parts of a dataset might be affecting your results. In the case of imaging models, you might explore different images based on factors like poor lighting or over exposure. We also recommend creating model cards, which can help explain any potential limitations, any trade-offs you have to make for performance, and then, providing a way to test out what the model does. Explainable AI methodsWhen we talk about explainable AI methods, it’s important to understand the difference between global and local methods.A global method is understanding the overall structure of how a model makes a decision. A local method is understanding how the model made decisions for a single instance. For instance, a global method might be that you look at a table that includes all the features that were used, ranked by the overall importance they have for making a decision. Feature importance tables are commonly used to explain structured data models to help people understand how specific input variables impact the final output of a model.But what about explaining how a model makes a decision for an individual prediction or a specific person? This is where local methods come into play. For the purpose of this post, we’ll cover local methods based on how they can be used for explaining model predictions in image data.Here are the most common explainable AI local methods:Local interpretable model-agnostic explanation (LIME)  Kernel Shapley additive explanations (KernalSHAP) Integrated gradients (IG) Explainable explanations through AI (XRAI) Both LIME and KernalShap break down an image into patches, which are randomly sampled from the prediction to create a number of perturbed (i.e. changed) images. The image will look like the original, but parts of the image have been zeroed out. Perturbed images are then fed to the trained model and asked to make a prediction. In the example below, the model would be asked: Is this image a frog or not a frog?The model would then provide the probability of whether the image is a frog. Based on the patches that were selected, you can actually rank the importance of each patch to the final probability. Both these methods can be used to help explain the local importance for determining whether the image contained a frog.Integrated gradients is a technique used to give importance value based on gradients of the final output. IG takes baseline images and compares them to the actual pixel value of the images that contain the information the corresponding model is designed to identify. The idea is that the value should improve in accuracy when the image contains what the model was trained to find. It helps determine how much a gradient changes from the baseline image to the point where it makes a prediction, providing an attribution mask that helps determine what the image is using to classify an image.XRAI is a technique that combines all of the three methods mentioned above, combining patch identification with integrated gradients to show salient regions that have the most impact on a decision, rather than individual pixels. The larger regions in this approach tend to deliver better results.Another emerging method that we’re starting to incorporate at Google Cloud is TracIn—a simple, scalable approach that estimates training data influence. The quality of ML model’s training data can have a huge impact on a model’s performance. TracIn tracks mislabeled examples and outliers from various datasets and helps explain predictions by assigning an influence score to each training example. If you are training a model to predict whether images have zucchinis, you would look at the gradient changes to determine which reduce loss (proponents) and increase loss (opponents). TracIn allows you to identify what images allow the model to learn to identify a zucchini and which are used to distinguish what’s not a zucchini.Using Explainable AI in Google CloudWe launched Vertex Explainable AI to help data scientists not only improve their models but provide insights that make them more accessible for decision-makers. Our aim is to provide a set of helpful tools and frameworks that can help data science teams in a number of ways, such as explaining how ML models reach a conclusion, debugging models, and combating bias.With Vertex Explainable AI platform, you can:Design interpretable and inclusive AI. Build AI systems from the ground up with Vertex Explainable AI tools designed to help detect and resolve bias, drift, and other gaps in data and models. With AI Explanations, data scientists can use AutoML Tables, Vertex Predictions, and Notebooks to explain how much a factor contributed to model predictions, helping to improve datasets and model architecture. The What-If Tool enables you to investigate model performance across a wide range of features, optimize strategies, and even manipulate individual datapoint values.Deploy ML models with confidence by providing human-friendly explanations. When deploying a model on AutoML Tables or Vertex AI , you can reflect patterns found in your training data to get a prediction and a score in real time about how different factors affected the final output. Streamline model governance with performance monitoring and training. You can easily monitor predictions and provide ground truth labels for prediction inputs with the continuous evaluation feature. Vertex Data Labelingcompares predictions with ground truth labels to incorporate feedback and optimize model performance.  AI continues to be an exciting frontier that will continue to shape and inspire the future of enterprises across all industries. But in order for AI to reach its full potential and gain wider adoption, it will require that all stakeholders, not just data scientists, understand how ML models work. That’s why, we remain committed to ensuring that no matter where AI goes in the future, it will serve everyone—be it customers, business users, or decision-makers. Next stepsLearn how to serve out explanations alongside predictions by running this Jupyter notebook on Cloud AI Platform. Step-by-step instructions are also available on Qwiklabs. And if you are interested in what’s coming in machine learning over the next five years, check out our Applied ML Summit to hear from Spotify, Google, Kaggle, Facebook and other leaders in the machine learning community.Related ArticleGoogle Cloud unveils Vertex AI, one platform, every ML tool you needGoogle Cloud launches Vertex AI, a managed platform for experimentation, versioning and deploying ML models into production.Read Article
Quelle: Google Cloud Platform

The top cloud capabilities industry leaders want for sustained innovation

Cloud computing technologies help companies and governments to deliver essential services to their customers and citizens—never was this seen more than during the pandemic. From enabling the quick rollout of indispensable programs like unemployment assistance or access to COVID-19 testing online portals to leveraging on-demand infrastructure to meet enterprise compute needs, cloud empowers IT leaders to react and respond quickly under extreme pressure.  With increasingly complex environments, which include a mix of proprietary and vendor solutions, legacy apps, geographically distributed, and resources living both on-premises or across multiple clouds, enterprises and agencies want to achieve more agility and improve cost efficiencies without getting locked into a single vendor in the future. At the same time, they are looking to leverage emerging technologies like edge solutions with the rollout of 5G.Multicloud and hybrid cloud approaches, coupled with open-source technology adoption, enable IT teams to take full advantage of the best cloud has to offer. And a recent study from the International Data Group (IDG) shows just how much of a priority this has become for business leaders. Multicloud and hybrid cloud capabilities among ‘must-haves’ from cloud providersAfter more than a year of uncertainty, organizations are applying lessons learned along the way as they assess the capabilities they need out of their cloud providers to keep pace with rapidly evolving requirements.  The results of the Google-commissioned study by IDG, based on a global survey of over 2,000 IT decision-makers, shows that multicloud/hybrid cloud support and other cutting edge technologies, such as containers, microservices, service mesh, and AI-powered analytics, are now major considerations for enterprises when selecting a cloud provider. This is true at almost all companies, regardless of their digital maturity, including those who are fully transformed (digital natives), currently implementing strategy (digital forwards), or not yet implementing any transformation strategy (digital conservatives).Related ArticleRead ArticleOrganizations are progressively more committed to the cloud, especially those who are further along on their digital transformation journey. The survey found that the majority of digital natives (83%) and digital forward (81%) companies list multicloud/hybrid cloud support and cutting-edge technology as key considerations when considering a cloud provider. However, the same factors are still among the top considerations for over 70% of digital conservatives.Another trend that goes hand-in-hand with multicloud and hybrid cloud support is the broader adoption of open source software solutions. In particular, open source technologies address barriers that arise from the need to modernize or integrate legacy systems and technologies—a primary pain point that impedes transformation efforts.  While once viewed as unconventional, open source has become vital to unlocking cloud innovation, delivering the speed and rich capabilities needed to speed production and increase creativity. This link between cloud and open source is also reflected in the IDG study results.  While globally, 74% of global IT leaders say they prefer open-source cloud solutions, this number jumps to 82% at digital-forward organizations and 87% for digital natives. By comparison, the same is true for just over half of digital-conservative companies.Freedom to innovate—anywhereGoogle Cloud’s commitment to multicloud, hybrid cloud, and open source enables our customers to use their data as well as build and run apps in the environment of their choice, whether on-premises, in Google Cloud, on another cloud provider, or across geographic regions. To learn more about the IDG findings and how IT leaders are creating new ways to operate and innovate, download the full report.Interested in how Google Cloud’s commitment to multi/hybrid cloud and OSS empowers transformation and drives innovation?Our distributed cloud services, including Anthos and Google Kubernetes Engine (GKE),  provide consistency between any public and private clouds as well as a solid foundation for modernization and future growth, while allowing developers to build, manage, and innovate faster, anywhere. Anthos extends Google Cloud’s best-in-breed solutions to any environment, enabling teams to modernize apps faster and establish operational consistency. It can be used for both legacy and cloud-native deployments, running on existing virtual machines (VMs) and bare metal services, while minimizing vendor lock-in and meeting regulatory requirements. Google Cloud’s commitment to multicloud, hybrid cloud, and open source enable organizations to leverage their data and run their applications and services on the cloud or in the environment of your choice, rather than using a single vendor solution. We aim to support our customers’ journeys to reinvention, and we hope that together we can pave the way for whatever is coming next.Related ArticleRead Article
Quelle: Google Cloud Platform