Kategorie: Google Cloud Platform

Reduce your cloud carbon footprint with new Active Assist recommendations

Last year, we analyzed the aggregate data from all customers across Google Cloud, and found over 600,000 gross kgCO2e in seemingly idle projects that could be cleaned up or reclaimed — which would have a similar impact to planting almost 10,000 trees1. Today, we’re making it easy for you to identify if any of those idle workloads are yours, with new Active Assist sustainability recommendations.  Active Assist is a part of Google Cloud’s AIOps solution that uses data, intelligence, and machine learning to reduce cloud complexity and administrative toil. Under the Active Assist portfolio, we have products and tools like Policy Intelligence, Network Intelligence Center, Predictive Autoscaler, and a collection of Recommendations for various Google Cloud services — all focused on helping you achieve your operational goals. Today, we are broadening the scope of Active Assist to help you achieve your sustainability targets and reduce the carbon footprint of your workloads.The carbon emissions associated with your cloud infrastructure can be a big part of your overall environmental footprint. Choosing to run on Google Cloud is a great first step — we’ve matched the energy used by our data centers with 100% renewable energy since 2017, and are committed to running our operations on carbon-free energy 24/7 by 2030. But once you’re running on Google Cloud, if you want to reduce the gross carbon emissions of your workload you can take action to optimize your usage.Assessing the gross carbon impact of unattended projectsYou can now estimate the gross carbon emissions you’ll save by removing these idle projects with Active Assist Unattended Project Recommender, which provides rich utilization insights for all the projects in your organization, and uses machine learning to identify ones that are idle and most likely unattended. The data points Active Assist surfaces as a part of its utilization insights now include the carbonFootprintDailyKgCO2 field, which allows you to estimate carbon emissions associated with any given project. Recommendations also estimate the impact of removing an idle project in terms of kilograms of CO2 reduced per month. The capability is available via the Recommender API, Recommendation Hub, the Carbon Footprint dashboard, and BigQuery export of recommendations, making it easy for you to integrate with your company’s existing tools and workflows.Example unattended project in Recommendation HubIntroducing the Carbon Sense suiteIncreasing the sustainability of digital applications and infrastructure is a priority for 90% of global IT leaders2, and we’ll be continuing to invest across a number of product areas in Google Cloud, including AIOps features like Active Assist’s recommendations, to help you make progress towards your sustainability goals. To make it easy for you to find and consume these new features, we’re bundling our existing and future product work into the Carbon Sense suite — a collection of features that makes it easy to accurately report your carbon emissions, and reduce them. Active Assist joins products like Carbon Footprint, which provides you with the ability to understand and measure the gross carbon emissions of your Google Cloud usage, and our low-carbon signals, which help users choose cleaner regions to run their workloads, in the Carbon Sense suite. Stay tuned for more updates on Carbon Sense in the coming months. Getting started with sustainability recommendationsTo get started with Active Assist sustainability recommendations, check the Carbon Footprint dashboard and Recommendation Hub to review projects that may be idle and assess the carbon emissions associated with them. See recommendations in Google Cloud Console.To view the recommendations, you will need IAM permissions for Unattended Project Recommender itself and permissions to view resources in a given organization.You can also automatically export the recommendations from your Organization to BigQuery and then investigate any idle projects with DataStudio or Looker. Or, you can use Connected Sheets to use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.As with any other Recommender, you can choose to opt out of data processing for your organization or your projects at any time by disabling the appropriate data groups in the Transparency & Control tab under Privacy & Security settings.We hope you use Unattended Project Recommender to reduce the carbon footprint associated with your idle cloud resources, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com. We also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to new features as they are developed.1. https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator2. https://inthecloud.withgoogle.com/it-leaders-research-21/sustainability-dl-cd.html
Quelle: Google Cloud Platform

Supercharge your event-driven architecture with new Cloud Functions (2nd gen)

Today, we are introducing Cloud Functions (2nd gen), Google Cloud’s next-generation Functions-as-a-Service product. This next generation version of Cloud Functions comes with an advanced feature set giving you more powerful infrastructure, advanced control over performance and scalability, more control around the functions runtime and triggers from over 90 event sources. Further, the infrastructure is powered by Google Cloud’s cutting-edge serverless and eventing infrastructure, Cloud Run and Eventarc.Infrastructure that meets your workloads’ needsCloud Functions adds a range of new capabilities for 2nd gen functions, such as concurrency (up to 1,000 concurrent requests per function instance), larger instances (16 GB memory and 4 vCPUs) and longer processing time for HTTP functions (up to 60 mins) and minimum instances (prewarmed instances). Longer request processing – Run your 2nd gen cloud functions for up to 60 mins for HTTP functions, making it easier to run longer-request workloads such as processing large streams of data from Cloud Storage or BigQuery.Larger instances – Leverage up to 16GB of RAM and 4 vCPUs on 2nd gen cloud functions, allowing larger in-memory, compute-intensive and more parallel workloads.Concurrency – Leverage up to 1000 concurrent requests with a single function, minimizing cold starts and improving latency and cost when scaling.Minimum instances – Provide for pre-warmed instances to cut your cold starts and make sure the bootstrap time of your application does not impact application performance.Traffic splitting – 2nd gen cloud functions support multiple revisions of your functions, splitting traffic between different revisions and rolling your function back to a prior version.Broader event coverage and CloudEvents support2nd gen cloud functions now include native support for Eventarc, which brings over 90+ event sources from direct sources and Cloud Audit logs (e.g., BigQuery, Cloud SQL, Cloud Storage, Firebase…). And of course, Cloud Functions still supports events from custom sources by publishing to Pub/Sub directly. These event-driven functions adhere to industry-standard CloudEvents, regardless of the source, to ensure a consistent developer experience.New developer experienceCloud Functions features an enhanced UI, customizable dashboard, improved developer experience and accessibility updates. A new seamless onboarding experience makes it easy to quickly develop and deploy your 1st gen and 2nd gen functions in one place. A deployment progress tracker navigates through the process of the 2nd gen function deployment and helps to spot the errors associated with each step. The UI also simplifies integrations with Eventarc using new menus and badges to help you find information about your function.Portability based on OSS buildpacks and Functions Frameworks2nd gen functions are built using open-source buildpacks and Functions Frameworks, giving you the portability to run your functions anywhere. Check out the new Cloud FunctionsWe are excited to see what you build with Cloud Functions. You can learn more about Cloud Functions here and get started using Quickstarts: Cloud Functions.Related ArticleNew Cloud Functions min instances reduces serverless cold startsSetting ‘min instances’ on your Cloud Functions applications translates to lower startup times.Read Article
Quelle: Google Cloud Platform

Data modernization with Google Cloud and MongoDB Atlas

What does modernization mean?As an IT leader or architect, you may notice that your software architecture is encountering performance issues. You may be considering moving your datastore from a mainframe or a traditional relational database (RDBMS) to a more modern database to take advantage of advanced analytics, scale at a faster rate, and opportunities to cut costs. Such is the impetus for modernization.An approach to modernization can be defined as, “An open, cross-functional collaboration dedicated to building new design systems and patterns that support evolving computing capabilities, information formats, and user needs.”Within the same spirit of modernization we can say that MongoDB works along with Google Cloud technologies to provide joint solutions and some reference architectures to help our customers leverage this partnership.Principles of modern technology solutionsA point of view to Modernization is understood through four basic principles that focus on outcomes for our customers. These principles can be applied to envision what a modern solution should achieve or to identify whether a given solution is modern or not.Help users get more done. Bring quality information forward and make it actionable in context. Actions are the new blue links.Feed curiosity. Open doorways to rich, endless discovery. Remove dead ends for users who want to engage more.Reflect the world, in real time. Surface fresh, dynamic content. Help users be in the know.Be personal, then personalize. Encourage the user’s personal touch to surface personal content and personalized experiences. Be stateful and contextual.Modern applications should be capable of presenting information in a way that enables users to not only make decisions, but also transform those decisions into actions. This requires the use of variable data formats and integration mechanisms that will allow the end user to interact with various systems and produce real-time results, without the need to log in to each one of them.MongoDB Atlas, a modern database management systemIf we are to use the four principles of modernization as a reference to identify modern solutions, then MongoDB Atlas reflects these directly. Altas helps database and infrastructure administrators get more done faster and with less effort than managing MongoDB on premises. It is a fully managed database service that takes care of the most critical and time-consuming tasks related to providing a continuous and reliable service, including security and compliance features out of the box, freeing administrators’ and developers’ time to focus on innovation.The third principle talks about reflecting the world in real time. This is the most cumbersome and daunting task for anybody who is responsible for the design of a modern technology system, since it requires an architecture capable of receiving, processing, storing, and producing results from data streams originated by different systems, at different velocity rates, and in different formats. Atlas frees the solution architect from this burden. As a managed service, it takes care of the networking, processing, and storage resources allocation, so it will scale as needed, when needed. And as a document-based database, it also allows for flexibility in regards to the format and organization of incoming data, Developers can focus on the actual process rather than spend their time modeling the information to make it fit into the RDBMS, as so often happens with traditional relational database schemas. It also provides real-time data processing features that allow for the execution of code or the consumption of external APIs residing in separate applications or even in various clouds.Of course, the combination of the first three principles leads to the fourth, which is to personalize the experience to the end user. Businesses must be able to solve specific user needs, rather than limit their processes solely to what their database or application is capable of. Putting the user first invariably leads to a better and modern experience—and that starts with choosing the best cloud provider and a database that aligns with these principles.A reference architecture for data modernizationLet’s dive into a general view of the migration reference architecture that enables the four aforementioned principles.An Operational Data Layer (or ODL) is an architectural pattern that centrally integrates and organizes siloed enterprise data, making it available to consuming applications. It enables a range of board-level strategic initiatives such as Legacy Modernization and Data as a Service, and use cases such as single view, real-time analytics and mainframe offload.An Operational Data Layer is an intermediary between existing data sources and consumers that need to access that data. An ODL deployed in front of legacy systems can enable new business initiatives and meet new requirements that the existing architecture can’t handle— without the difficulty and risk of a full rip and replace of legacy systems.For an initial migration that will keep the current architecture in place while replicating records that are produced over the production system, the following reference shows some components that can be taken into account to achieve a starting point in time backup and restore on MongoDB Atlas, while at the same time enabling real time synchronization.Figure 1. One-time data migration and real-time data syncThe above solution architecture shows both general views for one-time data migration and real-time data synchronization using Google Cloud technologies. A one-time data migration involves initial bulk ETL of data from the source relational database to MongoDB. Google Cloud Data Fusion can be used along with Apache Sqoop or Spark SQL’s JDBC connector powered by Dataproc to extract data from the source and store it in Google Cloud Storage temporarily. Custom Spark jobs powered by Dataproc are deployed to transform the data and load into MongoDB Atlas. MongoDB has a native spark connector which will allow storing Spark DataFrame as collections.Figure 2. One-time data migrationIn most of the migrations, the source database will not be retired for a few weeks to months. In such cases, MongoDB Atlas needs to be up to date with the source database. We can use Change Data Capture (CDC) tools like Google Cloud Datastream or Debezium on Dataflow to capture the changes, which can then be pushed to message queues like Google Cloud Pub/Sub. We can write custom transformation jobs using Apache beam powered by Dataflow, Java, or Python, which can consume the data from the message queue, transform it, and push it to MongoDB Atlas using native drivers. Google Cloud Composer will help orchestrate all the workflows.Figure 3. Real-time data synchronizationCommon use cases for MongoDBBelow are some observed common patterns of MongoDB. (For a more general treatment of more patterns please check out the MongoDB use case page.)Monolith to microservice – With its flexible schema and capabilities for redundancy, automation, and scalability, MongoDB (and MongoDB Atlas, its managed services version) is very well suited for microservices architecture. Together, MongoDB Atlas and microservices on Google Cloud can help organizations better align teams, innovate faster, and meet today’s demanding development and delivery requirements with full sharding across regions and globally.Legacy modernization – Relationship databases impose a tax on a business—a Data and Innovation Recurring Tax (DIRT). By modernizing with MongoDB, you can build new business functionality 3-5x faster, scale to millions of users wherever they are on the planet, and cut costs by 70% and more—all by unshackling yourself from legacy systems and, at the same time, taking advantage of the Google Cloud ecosystem. Mainframe offload – MongoDB can help offload key applications from the mainframe to a modern data platform without impacting your core systems, and help achieve agility while also reducing costs.Real-time analytics – MongoDB makes it easy to scale to the needs of real-time analytics with Atlas on Google Cloud; coupled with Google cloud analytics, such as BigQuery, the sky’s the limit.Mobile application development- MongoDB Realm helps companies build better apps faster with edge-to-cloud sync and fully managed backend services, including triggers, functions, and GraphQL.Other reference architecturesBelow are some reference architectures that can be applied to particular requirements. For more information, visit:MongoDB Use CasesGoogle Cloud Architecture CenterAn Operational Data Warehouse requires swift response times to keep data updated to the most recent state possible, with the final goal to produce near-real-time analytics. It also has to be scalable, robust, and secure to adapt to the highest standards and be compliant with various regulations.Figure 4. Operationalized Data Warehouse (ODS + EDW)The above referenced architecture describes which Google Cloud components can be combined to ingest data from any source into an ODS supported by MongoDB Atlas and how to integrate this ODS with an Enterprise Data Warehouse (BigQuery) that enables structured data for analytical tools like Looker.Shopping Cart AnalysisFigure 5 illustrates an implementation example of the Operationalized Data Warehouse reference architecture shown previously. In this scenario, several data sources (including shopping cart information) are replicated in real time to MongoDB through the Spark Connector. Information is then processed using Dataflow as a graphical interface to generate data processing jobs that are executed over an ephemeral, managed Hadoop & Spark cluster (Dataproc). Finally, processed data can be structured and stored for fast querying in BigQuery, supporting Shopping Cart, Product Browsing, and Outreach applications.Figure 5. Shopping cart analysisRecommendation EnginesFigure 6 presents a continuation of the idea presented in the last example. Now the objective is to use MongoDB Atlas as an Operational Data Warehouse that combines structured and semistructured data (SQL and noSQL data) in real time. This works as a centralized repository that enables machine learning tools such as Spark Mlib running on Dataproc, Cloud Machine Learning (now Vertex AI), and Prediction API to analyze data and produce personalized recommendations for customers visiting an online store in real time.Data from various systems can be ingested as-is and stored and indexed in JSON format in MongoDB. Dataproc would then use MongoDB Apache Spark Connector to perform the analysis. The insight would be stored in BigQuery and distributed to applications downstream.Learn more about MongoDB and Google Cloud at cloud.google.com/mongodbRelated ArticleLooker lets you choose what works best for your dataEmbrace platform freedom with Looker. Learn about how we are expanding our features as a cloud platform to meet the unique needs of every…Read Article
Quelle: Google Cloud Platform

Cloud Scheduler: Now available across 23 GCP Regions

Reliably executing tasks on a schedule is critical for everything from data engineering, to infrastructure management, and application maintenance. Today, we are thrilled to announce that Google Cloud Scheduler, our enterprise-grade scheduling service, is now available in more GCP regions and multiple regions can now be used from a single project removing the prior limit of a single region per project.With many enterprise customers deploying complex distributed cloud systems, Cloud Scheduler has helped solve the problem of single-server cron scheduling being a single point of failure. With this update you are now able to create Scheduler jobs across distinct cloud regions that can help satisfy cross-regional availability and fail-over scenarios. Furthermore, you are no longer required to create an AppEngine application in order to use Cloud Scheduler. For existing Cloud Scheduler jobs, it is safe to disable the AppEngine application within the project. Jobs will continue to function without an AppEngine application. Creating jobs in different regions is easy. You simply pick the location where you would like your job to run. For example you can specify a location when creating a job through the gcloud command line :HTTP TargetsPub/Sub TopicsAppEngine ServicesOr you can pick a location when creating a job through the Cloud Console:Google Cloud Scheduler is now available in 23 GCP Regions,  and this number is expected to grow in the future. You can always find an up-to-date list of available regions by running:We hope you are as excited about this launch as we are. Please reach out to us with any suggestions or questions in our public issue tracker.Related ArticleCloud Tasks: Now available in 23 GCP RegionsLaunch announcement for Google Cloud Tasks service availability in 23 new GCP RegionsRead Article
Quelle: Google Cloud Platform

Introducing Compute Optimized VMs powered by AMD EPYC processors

Over the last six months, we launched 3rd Gen AMD EPYC™ CPUs (formerly code-named “Milan”) across our Compute Engine virtual machine (VM) families. We introduced the Tau VM family, targeting scale-out workloads. Tau VMs are the leader both in terms of performance and workload total cost of ownership (TCO) from any leading provider available today. We also refreshed our general-purpose N2D instances with 3rd Gen AMD EPYC processors, providing a 30% boost in price-performance. Today, we’re excited to announce the General Availability of the newest instance series in our Compute Optimized family, C2D, also powered by 3rd Gen AMD EPYC processors.”AMD EPYC processors continue to showcase their capabilities for HPC and compute-focused workloads. Whether that’s running drug simulations for the latest vaccines, exploring the cosmos, or helping design critical hardware and electronics for the future of the industry,” said Lynn Comp, corporate vice president, Cloud Business, AMD. “The Google Cloud C2D instances with AMD EPYC processors show the continued growth of the AMD and Google Cloud collaboration, by now offering some of the highest performance instances for demanding, performance-intensive workloads.”New larger machine shapes for the Compute Optimized FamilyC2D instances take advantage of advances in processor architecture from the latest generation AMD EPYC™ CPUs including  “Zen 3” core.  C2D supports Persistent Disks, Advanced Networking, Compact Placement Policies, and soon-to-follow Sole Tenant nodes. Instances are configurable with up to 112 vCPUs (56 cores), 896 GB of memory, and 3 TB of Local SSD. C2D is available in standard, high-cpu and high-mem, each with seven machine types for optimal memory-to-core ratio, to better align with your workload. Improved performance for a wide variety of workloads The Compute Optimized VM family is ideal for customers with performance-intensive workloads. C2D instances provide the largest VM sizes within the Compute Optimized VM family and are best-suited for memory-bound workloads such as high-performance databases, gaming, and high-performance computing (HPC) workloads, such as electronic design automation (EDA) and computational fluid dynamics (CFD). C2D high-cpu and standard instances serve existing compute-intensive workloads, including high-performance web servers, media transcoding, and AAA Gaming. C2D high-mem machine configurations are well suited for workloads such as HPC and EDA that require higher memory configurations. For optimal HPC workload performance, check out Google’s best practices for running tightly-coupled HPC applications on Compute Engine.Performance reportWe’ve illustrated below how C2D with 3rd Gen EPYC compares against N2D with 2nd Gen EPYC (formerly code-named “Rome”)    in GCP’s preferred set of benchmarks to measure compute intensive performance, media transcoding, and gaming benchmarks.We worked with AMD engineers to benchmark some key applications in the HPC industry. The improvements in the Compute Optimized family are clear when C2D is compared directly to AMD’s previous generation of EPYC processors, specifically the n2d-standard-128 machine shape, closest to C2D’s 112 vCPUs. We first compare performance on industry-standard measures of memory bandwidth (STREAM Triad) and floating-point performance (HPL).Compared to the N2D VM’s baseline performance, the C2D’s 3rd Gen EPYC processor improvements, including higher L3 cache sizes per core and full NUMA exposure, have a direct benefit in memory performance. This is empirically observed through the 30% improved STREAM Triad results. C2D’s floating-point improvements can also be seen in the 7% performance increase in the HPL results, despite being run with 12.5% fewer cores than the previous-generation EPYC processor. Looking at application benchmarks across some key areas of focus in HPC, we can see that C2D VMs provide material gains for representative benchmarks in areas such as weather forecasting (WRF CONUS 2.5km), molecular dynamics (NAMD), and CFD (OpenFOAM).What customers are sayingNot only is the c2d-standard-112 machine shape faster overall in the above workloads, but it’s also ~6% cheaper than the baseline n2d-standard-128 machine shape. It’s no wonder that customers are choosing it for their memory-intensive and HPC workloads. Here’s a sampling.AirShaper is an cloud-based CFD platform that helps designers and engineers to easily run aerodynamic simulations to improve the performance and efficiency of cars, drones, motorbikes — even athletes themselves.“Getting the best performance helps us drastically reduce run times, improving user experience and cutting costs at the same time. By running our CFD simulations on C2D, we’ve been able to reduce our costs by almost 50% and reduce simulation times by 30% compared to previous generation high-performance computing instances. Also, compared to our on-prem instances we’ve been able to reduce our simulation times by more than a factor of three.” – Wouter Remmerie, CEO AirshaperClutch’s Integrated Customer Data and Marketing platform delivers customer intelligence and personalized engagements for brands to identify, understand and motivate each segment of their customer base. Clutch offers solutions for CDP, Loyalty, Offer Management, Marketing Orchestration and Stored Value that use embedded machine learning to increase the lifetime value of each customer.“We moved our compute and memory intensive Data Analytics platform to Compute Optimized on AMD EYPC Milan instances. The C2D instances provide a sweet spot of memory and CPU performance.” – Ed Dunkelberger, SVP TechnologyGoogle Kubernetes Engine supportGoogle Kubernetes Engine (GKE) is the leading platform for organizations looking for advanced container orchestration, delivering the highest levels of reliability, security, and scalability. GKE supports C2D VMs, helping you get the most out of your containerized workloads. You can add C2D 3rd Gen EPYC CPU-based VMs to your GKE clusters by choosing the C2D machine type in your GKE node pools. Confidential Computing (coming soon)Confidential Computing is an industry-wide effort to protect data in-use including encryption of data in-memory — while it’s being processed. With Confidential Computing, you can run your most sensitive applications and services on C2D VMs.We’re committed to delivering a portfolio of Confidential Computing VM instances and services such as GKE and Dataproc using the AMD Secure Encrypted Virtualization (SEV) security feature. We’ll support SEV using this latest generation of AMD EPYC™ processors in the near term and plan to add more security capabilities in the future.Get started with C2D todayC2D instances are available today in regions around the globe: us-central1 (Iowa), asia-southeast1 (Singapore), us-east1 (South Carolina), us-east4 (North Virginia), asia-east1 (Taiwan), and europe-west4 (Netherlands), and in additional regions in the coming months. C2D instances are available via on-demand, as Spot VMs, and via reservations. You can also take advantage of further cost savings by purchasing Committed Use Discounts (CUDs) in one- and three-year terms. To start using C2D instances, simply choose the C2D option when creating a new VM or GKE node in the Google Cloud Console.Related ArticleCompute Engine explained: Choosing the right machine family and typeAn overview of Google Compute Engine machine families and machine types.Read Article
Quelle: Google Cloud Platform

3 ways Dataflow is delivering 50%+ productivity boost and cost savings to customers

In our conversations with technology leaders about data-driven transformation using Google Data Cloud –  industry’s leading unified data and AI solution – , one important topic is incorporating continuous intelligence to move from answering questions such as “What has happened? to questions like “What is happening?” and “What might happen?”. The core to this evolution is the need for an underlying data processing that not only provides powerful real-time capabilities for events happening close to origination, but also brings together existing data sources under one unified data platform to enable organizations to draw insights and take actions holistically. Dataflow, Google’s cloud-native data processing and streaming analytics platform, is a key component of any modern data and AI architecture and data transformation journey, along with BigQuery, Google’s internet-scale warehouse with built-in streaming, BI engine and ML; Pub/Sub, a global no-ops event delivery service; and Looker, a modern BI and embedded analytics platform. One of the key evaluation factors is potential economic value of Dataflow to their organization, particularly in the context of engaging other stakeholders is key for many of the leaders that we engage with. So we commissioned Forrester Consulting to conduct a comprehensive study on the impact that Dataflow had on their organization by interviewing actual customers . Today we’re excited to share our commissioned study conducted by Forrester Consulting, the Total Economic Impact™ of Google Cloud Dataflow, which allows data leaders to understand and quantify the benefits of Dataflow, and use cases it enables. Forrester conducted interviews with Dataflow customers to evaluate the benefits, costs, and risks of investing in Dataflow across an organization. Based on their interviews, Forrester identified major financial benefits across four different areas: business growth, infrastructure cost savings, data engineer productivity, and administration efficiency. In fact, Forrester found that customers adopting Dataflow can achieve a 55% boost in developer productivity and a 50% reduction in infrastructure costs. In fact, Forrester projects that customers adopting Dataflow can achieve a range of up to 171% Return on Investment (ROI) and a less than six months payback period. Customers can now use figures in the report to compute their own Return on Investment (ROI) and payback period.“Dataflow is integral to accelerating time-to-market, decreasing time-to-production, reducing time to figure out how to use data for use cases, focusing time on value-add tasks, streamlining ingestion, and reducing total cost of ownership.” – Lead technical architect, CPGLet’s take a deeper look at the ways that Forrester found that Dataflow can help you achieve your goals and unlock your business potential. Benefit #1: Increase data engineer productivity by 55%Developers can choose among a variety of programming languages to define and execute data workflows. Dataflow also seamlessly integrates with other Google Cloud Platform and open source technologies to maximize value and applicability to a wide variety of use cases. Dataflow streamlined workflows with code reusability,dynamic templates, and the simplicity of a managed service. Engineers trusted pipelines to run correctly and adhere to governance. Data engineers avoided laborious issue-monitoring and remediation tasks that were common in the legacy environments such as poor performance, lack of availability, and failed jobs. Teams valued the language flexibility and open source base.“Dataflow provided us with ETL replacement that opened limitless potential use cases and enabled us to do smarter data enhancement while data remains in motion.” — Director of data projects, financial servicesBenefit #2: Reduce infrastructure costs by up-to 50% for batch and streaming workloads Dataflow’s serverless autoscaling and discrete control of job needs, scheduling, and regions eliminated overhead and optimized technology spending. Consolidating global data processing solutions to Dataflow further eliminated excess costs while ensuring performance, resilience, and governance across environments. Dataflow’s unified streaming and batch data platform gives organizations the flexibility to define either workload in the same programming model, run it on the same infrastructure, and manage it from a single operational management tool. “Our costs with our cloud data platform using Dataflow are just a fraction of the costs we faced before. Now we only pay for cloud infrastructure consumption because the open source base helps us avoid licensing costs. We spend about $120,000 per year with Dataflow, but we’d be spending millions with our old technologies.” – Lead technical architect, CPGBenefit #3: Increase top-line revenue by improving customer experience and retention with payback time of < 6 monthsStreaming analytics is an essential capability in today’s digital world to gain real-time actionable insights. Likewise, organizations must also have flexible, high- performance batch environments to analyze historical data for building machine learning models, business intelligence, and advanced analytics. Dataflow enabled real-time streaming use cases, improved data enrichment, encouraged data exploration,improved performance and resiliency, reduced errors, increased trust, and eliminated barriers to scale. As a result, organizations provided customers with more accurate, relevant, and in-the-moment data-backed services and insights — boosting customer experience, creating new revenue streams, and improving acquisition, retention, and enrichment.“It’s already been proven that we are getting more business [with Dataflow] because we can turn around results faster for customers.” – VP of technology, financial services technology“When we provide data to our customers and partners with Dataflow, we are much more confident in those numbers and can provide accurate data within a minute. Our customers and partners have taken note and commented on this. It’s reduced complaints and prevented churn.” – Senior software engineer, mediaOther benefits Eliminated administrative overhead and toilAs a cloud-native managed service, all administration tasks such as provisioning, scaling, and updates are automatically handled by Google Cloud. Teams no longer need to manage servers and related software for legacy data processing solutions. Admins also streamlined processes for setting up data sources, adding pipelines, and enforcing governance.Saved business operations costs for support teams and data end usersDataflow improved the speed, quality, reliability, and ease of access to data for insights for general business users, saving time and empowering users to drive better data-backed outcomes. It also reduced support inquiry volume while automating manual job creation.What’s next?Download the Forrester Total Economic Impact study today to dive deep into the economic impact Dataflow can deliver your organization. We would love to partner with you to explore the potential Dataflow can unlock in your teams. Please reach out to our sales team to start a conversation about your data transformation with Google Cloud.Related ArticleDataflow Prime: bring unparalleled efficiency and radical simplicity to big data processingCreate even better data pipelines with Dataflow Prime, coming to Preview in Q3 2021.Read Article
Quelle: Google Cloud Platform

Know more, spend less: how GKE cost optimization insights help you optimize Kubernetes

If there’s one thing we learned talking to Kubernetes users, it’s that  optimizing for reliability, performance, and cost efficiency is hard — especially at scale.That is why, not long ago, we released GKE cost optimization insights in preview, a tab within the Google Cloud Console that helps you discover optimization opportunities at scale, across your Google Kubernetes Engine clusters and workloads, automatically with minimal friction.The functionality allows you to figure out, over a selected period of time, the current state of your clusters by exposing the actual used, requested and allocated resources. For workloads running on your clusters, it shows your actual used and requested resources, as well as the set limits, so you can  make granular, workload-level right-sizing optimizations.GKE cost optimization insights have proved popular with users right out of the gate. For example, Arne Claus, Site Reliability Engineer at hotel search platform provider Trivago says that “The new GKE cost optimization insights view helped us to identify cost optimization opportunities at the cluster and workload level and take immediate action. In the first weeks of use, the Trivago team spotted and improved the cost/performance balance of several clusters.”Today, we’re graduating GKE cost optimization insights from Preview to GA. It has undergone multiple improvements that we believe will help you with your day-to-day optimization routines. For instance, we’ve made it easier to spot under-provisioned workloads that could be at risk of instability due to insufficient resource requests.Now that you have the insights into optimization opportunities, let’s recap what capabilities are helping the most with reliability, performance, and cost efficiency in GKE, and what resources are available for your teams to get up to speed with GKE cost optimization.In public cloud managed Kubernetes services, there are four major pitfalls that lead to non-optimized usage of Kubernetes clusters:Culture – Many teams that embrace the public cloud have never worked with a pay-as-you-go service like GKE before, so they’re unfamiliar with how resource allocation and app deployment processes can affect their costs. The new GKE cost optimization insights can help teams better understand such an environment and can help improve business value by providing insights into balancing cost, reliability and performance needs. Bin packing – The more efficiently you pack apps into nodes, the more you save. You can pack apps into nodes efficiently by ensuring you’re requesting the right amount of resources based on your actual utilization. GKE cost optimization insights helps you identify bin-packing gaps by looking at the gray bar in the cluster view.App right-sizing – You need to be able to configure the appropriate resource requests and workload autoscale targets for the objects that are deployed in your cluster. The more precise you are in setting accurate resource amounts to your pods, the more reliably your apps will run and, in the majority of cases, the more space you will open in the cluster. With GKE cost optimization insights, you can visualize the right-sizing information by looking at the green bar in both cluster and workload views.Demand-based downscaling – To save money during low-demand periods such as nighttime, your clusters should be able to scale down with demand. However, in some cases, you can’t scale them down because there are workloads that cannot be evicted or because a cluster has been misconfigured.GKE cost optimization insights help you better understand and visualize these pitfalls. In order to solve them, or make them non-issues right from the beginning, there are de-facto solutions available from Google Cloud. For example, you can use the new GKE cost optimization insights to help with monitoring and with the cultural shift toward FinOps. If you don’t want to deal with bin packing, you can use the Autopilot mode of operation. Set up node auto-provisioning along with optimize-utilization profile can also help optimize bin packing. To help with app right-sizing and demand-based downscaling you can take advantage of GKE Pod autoscalers — in addition to the classic Horizontal Pod Autoscaler, we also provide a Vertical Pod Autoscaler and a Multidimensional Pod Autoscaler.We’ve extensively written about GKE features such as Autopilot, optimized VM types, Node auto-provisioning, pod autoscalers and others in our GKE best practices to lessen overprovisioning. This is a great place to learn how to solve for your newly discovered optimization opportunities.If you want a deeper dive into technical details, check out these best practices for running cost-optimized Kubernetes applications on GKE, an exhaustive list of GKE best practices.And finally, for the visual learner, there’s the GKE cost optimization video series on Youtube, where our experts will walk you through key concepts of cost optimization step by step.Related ArticleFind your GKE cost optimization opportunities right in the consoleNew GKE cost optimization insights now appear in the console, making it easier to adopt Kubernetes best practices like app right-sizing a…Read Article
Quelle: Google Cloud Platform

Protecting customers against cryptomining threats with VM Threat Detection in Security Command Center

As organizations move to the cloud, VM-based architectures continue to make up a significant portion of compute-centric workloads. To help ensure strong protection for these deployments, we are thrilled to announce a public preview of our newest layer of threat detection in Security Command Center (SCC): Virtual Machine Threat Detection (VMTD). VMTD is a first-to-market detection capability from a major cloud provider that provides agentless memory scanning to help detect threats like cryptomining malware inside your virtual machines running in Google Cloud.The economy of scale enabled by the cloud can help fundamentally change the way security is executed for any business operating in today’s threat landscape. As more companies adopt cloud technologies, security solutions built into cloud platforms help address emerging threats for more and more organizations. For example, in the latest Google Cybersecurity Action Team Threat Horizons Report, we saw 86% of compromised cloud instances were used to perform cryptocurrency mining. VMTD is one of the ways we protect our Google Cloud Platform customers against growing attacks like coin mining, data exfiltration, and ransomware.Our unique approach with agentless VM threat detectionTraditional endpoint security relies on deploying software agents inside a guest virtual machine to gather signals and telemetry to inform runtime threat detection. But as is the case in many other areas of infrastructure security, cloud technology offers the ability to rethink existing models. For Compute Engine, we wanted to see if we could collect signals to aid in threat detection without requiring our customers to run additional software. Not running an agent inside of their instance means less performance impact, lowered operational burden for agent deployment and management, and exposing less attack surface to potential adversaries. What we learned is that we could instrument the hypervisor — the software that runs underneath and orchestrates our customers’ virtual machines — to include nearly universal and hard-to-tamper-with threat detection.Illustrative data path for Virtual Machine Threat DetectionGetting Started with Virtual Machine Threat Detection (VMTD)We’re excited about the kinds of detection that are possible with VMTD. During our public preview, VMTD detects cryptomining attacks. Over the next months as we move VMTD towards general availability, you can expect to see a steady release of new detective capabilities and integrations with other parts of Google Cloud. To get started with VMTD, open the Settings page in Security Command Center. Click on “MANAGE SETTINGS” under Virtual Machine Threat Detection. You can then select a scope for VMTD. To confirm that VMTD is working for your environment, you can download and execute this test binary that simulates cryptomining activity. Safeguarding customer trustWe know safeguarding users’ trust in Google Cloud is as important as securing their workloads. We are taking several steps to ensure the ways in which VMTD inspects workloads for potential threats preserves trust: First, we are introducing VMTD’s public preview as an opt-in service for our Security Command Center Premium customers. Additionally, not only does Confidential Computing provide encryption for memory as it moves out of a CPU to RAM, we never process memory in VMTD from Confidential nodes. Comprehensive threat detection with SCC PremiumVirtual Machine Threat Detection is fully integrated and available through Security Command Center Premium. VMTD complements the existing threat detection capabilities enabled by the Event Threat Detection and Container Threat Detection built-in services in SCC Premium. Together, these three layers of advanced defense provide holistic protection for workloads running in Google Cloud: Multiple layers of threat detection in Security Command CenterIn addition to threat detection, the premium version of Security Command Center is a comprehensive security and risk management platform for Google Cloud. It provides built-in services that enable you to gain visibility into your cloud assets, discover misconfigurations and vulnerabilities in your resources, and help maintain compliance based on industry standards and benchmarks.To enable a Security Command Center Premium subscription, contact your Google Cloud Platform sales team. You can learn more about all these new capabilities in SCC in ourproduct documentation.Related ArticleHow Vuclip safeguards its cloud environment across 100+ projects with Security Command CenterLearn how Security Command Center enables Vuclip to manage security and risk for their cloud environment.Read Article
Quelle: Google Cloud Platform