Kategorie: Google Cloud Platform

Opening doors, embracing change with cloud data warehouses

Moving from a traditional on-prem data warehouse platform to a cloud data warehouse, such as Google Cloud’s BigQuery, is about more than just adopting new technology. It also provides an opportunity to revisit existing practices and adopt new ways of working. We’ve noted some typical changes that teams undergo to support data warehouse migration to the cloud. To get a framework for change management and best practices for moving to the cloud, you can find more details in the Managing Change in the Cloud whitepaper.People form the core of any change management initiative. For a data warehouse migration initiative, we’ll focus on three key stakeholders who are involved in day-to-day usage and management (though there may be other ancillary roles involved): Data consumer—typically a data analyst, data scientist, or business user Data enabler—typically a data engineer or ETL developer Data administrator—typically a database/data warehouse administratorClick to enlargeLet’s take a look at each stakeholder and see how their responsibilities change as their organization transitions from on-prem to cloud. Data consumer Data analysts and data scientists have a lot to gain from migrating to a cloud data warehouse— easier availability of new datasets, new algorithms to play with, and lower-latency access options in the cloud. Most often, during the migration, existing reporting applications and BI tools are initially kept as is to ensure minimal business disruption—making it relatively easy for data consumers to adopt the changes. Data consumers, however, will likely want to ensure that their existing reports and use cases are tested on the new platform. It is, therefore, advisable to require their participation in any data validation efforts. You might create a data validation team, for example. Post-migration, data consumers can find new ways of working once they have access to new datasets, like converting batch reporting into real-time dashboards, and finding ways to make machine learning a reality for the organization. New platform capabilities end up becoming the most important catalyst for change in this role. The more ambitious the data consumers are, the greater the change will be for other roles in the organization.  Data enablerThe technical abilities of the data enabler are vital to how successfully cloud is adopted into the organization. These individuals own the data pipelines and are deeply involved in any reengineering efforts needed to migrate workloads to the cloud. The learning curve can be steep depending on the technology stack that’s adopted; proper resource planning will be crucial for these data enablers. Migration to the cloud in and of itself could be a long-term undertaking, so it’s worth considering taking the time to plan, re-tool and automate the migration. As data enablers are refactoring and redesigning data pipelines and working together with data consumers, they have plenty of opportunities to rethink how business processes should change to take advantage of real-time data ingestion, new data modeling techniques, or new persistent stores that cloud technologies offer. Data enablers might also revisit old asks that were not possible to deliver due to scale, data format complexity, or ETL complexity. Cloud technologies can be better suited to overcoming such challenges through capabilities such as storage and compute resource elasticity, support for numerous data format and storage options, as well as a wide variety of tools and libraries to process data. Beyond that, adopting a DevOps model, application containerization, or a serverless compute model are avenues that data enablers (along with data administrators) can explore to improve the speed with which changes are applied and new insights are delivered to users.Data administrator Database administrators continue to play a vital role even if the enterprise data warehouse is hosted in the cloud. As owners of the platform, administrators must get acquainted with cloud technology capabilities at the earliest and drive the enterprise transformation with key stakeholders. As database administrators move data workloads into the cloud, there continue to be important focus areas: governance overview during and post migration, data integrity, and SLAs such as RPOs and RTOs. If the organization adopts a vendor-managed cloud data warehouse, database administrators will have fewer platform procurement and database-tuning tasks. But cost vs. performance optimization is still a critical area for administrators to be fully involved in. (Check out ESG’s report on how migrating to BigQuery can lower your three-year TCO up to 52%.) Database administrators are also freed up to gain a deeper understanding of the storage, network, and compute options that cloud offers and look for ways to optimize enterprise workloads in the cloud. Administrators can adopt new ways of working with machine learning notebook environments, new query tools, containerization, hybrid cloud, and more—there’s plenty to learn and optimize for! Organizations that adopt these new skills and processes have a greater chance of success in delivering business results in the cloud. For more information on how to streamline your enterprise migration path, check out our data warehouse migration guide and framework. You can also apply for our data warehouse migration offer for help with the full migration process.
Quelle: Google Cloud Platform

Kubernetes Podcast in 2019: year-end recap

At the Kubernetes Podcast, we bring you a weekly round-up of cloud-native news, accompanied by an in-depth interview with a community member. As we publish our 50th and final episode for 2019, it’s time to look back on some of our favorite moments from the year.This year, we stepped out of the studio. We hosted a live recording at Google Cloud Next in San Francisco, as well as listener meetups at KubeCon EU in Barcelona and KubeCon NA in San Diego. There’s nothing more gratifying to us than having someone come up to you at a conference and tell you that they enjoy your show, or even ask after the family of foxes you mentioned were living in your backyard.  Our heartfelt thanks to everyone who came by, or stopped us in the hallways.Open source reaches all corners of the world, and we’ve been amazed at all the listeners who have joined the podcast community from around the globe. Every now and then we send out stickers by post: they’ve gone to dozens of countries on almost every continent. (We’re still waiting for a listener to reach out from Antarctica!) Thank you to our wonderful audience, who has let us know how much we’re helping them connect with and learn about the Kubernetes community. We are truly grateful to you for listening.Serious dedication tweeted to us from one podcast listenerWe would like to share some of our most popular episodes from 2019:Kubernetes Failure Stories, with Henning Jacobs (episode 38): To have the best chance for success, it helps to learn from failures. After experiencing some of his own, Henning was inspired to start collecting the failure stories of others.Ingress, with Tim Hockin (episode 41): A proud parent of the Kubernetes project, Tim is a 15-year Googler and designer of large parts of the Kubernetes networking and storage stack—an obvious extension of his years of work on the Linux kernel.Live at Google Cloud Next, with Eric Brewer (episode 49): In our first live show, Eric joined us to talk about his history in building infrastructure for search, the CAP theorem, and announcing Kubernetes to the world.KeyBank, with Gabe Jaynes (episode 51): Banks aren’t always terminals and mainframes. The smart ones, like KeyBank, are Kubernetes and mainframes! Gabe’s team worked with Google Cloud as a design partner.Istio 1.2, with Louis Ryan (episode 58): Louis has been working on API infrastructure and service mesh at Google for 10 years. He talked about the history of Istio, its design decisions, and its future goals.Attacking and Defending Kubernetes, with Ian Coldwater (episode 65): Learn how to protect your container infrastructure from Ian: they are paid to attack it, and a popular conference speaker on the topic.CRDs, API Machinery and Extensibility, with Daniel Smith (episode 73): Another long-time Kubernetes contributor, Daniel joined the project before it was open-sourced, and leads both the open-source and Google teams who build CRDs and other extensibility features.Kubernetes 1.17, with Guinevere Saenger (episode 83): Our penultimate episode for the year is an interview with the Release Team lead for the new Kubernetes 1.17. Learn how Guinevere went from being a concert pianist to a software engineer and leading a team of over 30 to produce the final Kubernetes release of 2019.If you have a break over the holidays, why not subscribe and enjoy one episode or many?  For those who can’t listen, or prefer not to, we also offer a transcript of each episode on its page at kubernetespodcast.com.We’re going to take a two-week break over the holiday period, but we’ll be back in your ears in January!
Quelle: Google Cloud Platform

5 best practices for Compute Engine Cost Optimization

When customers migrate to Google Cloud Platform (GCP), their first step is often to adopt Compute Engine, which makes it easy to procure and set up virtual machines (VMs) in the cloud that provide large amounts of computing power. Launched in 2012, Compute Engine offers multiple machine types, many innovative features, and is available in 20 regions and 61 zones! Compute Engine’s predefined and custom machine types make it easy to choose VMs closest to your on-premises infrastructure, accelerating the workload migration process cost effectively. Cloud allows you the pricing advantage of ‘pay as you go’ and also provides significant savings as you use more compute with Sustained Use Discounts. As Technical Account Managers, we work with large enterprise customers to analyze their monthly spend and recommend optimization opportunities. In this blog, we will share the top recommendations that we’ve developed based on our collective experience working with GCP customers. Getting ready to saveBefore you get started, be sure to familiarize yourself with the VM instance pricing page—required reading for anyone who needs to understand the Compute Engine billing model and resource-based pricing. In addition to those topics, you’ll also find information about the various Compute Engine machine types, committed use discounts and how to view your usage, among other things. Another important step to gain visibility into your Compute Engine cost is using Billing reports in the Google Cloud Console and customizing your views based on filtering and grouping by projects, labels and more. From there you can export Compute Engine usage details to BigQuery for more granular analysis. This allows you to query the datastore to understand your project’s vCPU usage trends and how many vCPUs can be reclaimed. If you have defined thresholds for the number of cores per project, usage trends can help you spot anomalies and take proactive actions. These actions could be rightsizing the VMs or reclaiming idle VMs.Now, with these things under your belt, let’s go over the five ways you can optimize your Compute Engine resources that we believe will give you the most immediate benefit. 1. Apply Compute Engine rightsizing recommendationsCompute Engine’s rightsizing recommendations feature provides machine type recommendations that are generated automatically based on system metrics gathered by Stackdriver Monitoring over the past eight days. Use these recommendations to resize your instance’s machine type to more efficiently use the instance’s resources. It also recommends custom machine types when appropropriate. Compute Engine makes viewing, resizing and other actions easier right from the Cloud Console as shown below. Recently, we expanded Compute Engine rightsizing capabilities from just individual instances to managed instance groups as well. Check out the documentation for more details.For more precise recommendations, you can install the Stackdriver Monitoring agent which collects additional disk, CPU, network, and process metrics from your VM instances to better estimate your resource requirements. You can also leverage the Recommender API for managing recommendations at scale.2. Purchase CommitmentsOur customers have diverse workloads running on Google Cloud with differing availability requirements. Many customers follow a 70/30 rule when it comes to managing their VM fleet—they have constant year-round usage of ~70%, and a seasonal burst of ~30% during holidays or special events. If this sounds like you, you are probably provisioning resources for peak capacity. However, after migrating to Google Cloud, you can baseline your usage and take advantage of deeper discounts for Compute workloads. Committed Use Discounts are ideal if you have a predictable steady-state workload as you can purchase a one or three year commitment in exchange for a substantial discount on your VM usage.We recently released a Committed Use Discount analysis report in the Cloud Console that helps you understand and analyze the effectiveness of the commitments you’ve purchased. In addition to this, large enterprise customers can work with their Technical Account Managers who can help manage their commitment purchases and work proactively with them to increase Committed Use Discount coverage and utilization to maximize their savings.3. Automate cost optimizationsThe best way to make sure that your team is always following cost-optimization best practices is to automate them, reducing manual intervention.Automation is greatly simplified using a label—a key-value pair applied to various Google Cloud services. For example, you could label instances that only developers use during business hours with “env: development.” You could then use Cloud Scheduler to schedule a serverless Cloud Function to shut them down over the weekend or after business hours and then restart them when needed. Here is an architecture diagram and code samples that you can use to do this yourself. Using Cloud Functions to automate the cleanup of other Compute Engine resources can also save you a lot of time and money. For example, customers often forget about unattached (orphaned) persistent disk, or unused IP addresses. These accrue costs, even if they are not attached to a virtual machine instance. VMs with the “deletion rule” option set to “keep disk” retain persistent disks even after the VM is deleted. That’s great if you need to save the data on that disk for a later time, but those orphaned persistent disks can add up quickly and are often forgotten! There is a Google Cloud Solutions article that describes the architecture and sample code for using Cloud Functions, Cloud Scheduler, and Stackdriver to automatically look for these orphaned disks, take a snapshot of them, and remove them. This solution can be used as a blueprint for other cost automations such as cleaning up unused IP addresses, or stopping idle VMs. 4. Use preemptible VMsIf you have workloads that are fault tolerant, like HPC, big data, media transcoding, CI/CD pipelines or stateless web applications, using preemptible VMs to batch-process them can provide massive cost savings. In fact, customer Descartes Labs reduced their analysis costs by more than 70% by using preemptible VMs to process satellite imagery and help businesses and governments predict global food supplies.Preemptible VMs are short lived— they can only run a maximum of 24 hours, and they may be shut down before the 24 hour mark as well. A 30-second preemption notice is sent to the instance when a VM needs to be reclaimed, and you can use a shutdown script to clean up in that 30-second period. Be sure to fully review the full list of stipulations when considering preemptible VMs for your workload. All machine types are available as preemptible VMs, and you can launch one simply by adding “-preemptible” to the gcloud command line or selecting the option from the Cloud Console. Using preemptible VMs in your architecture is a great way to scale compute at a discounted rate, but you need to be sure that the workload can handle the potential interruptions if the VM needs to be reclaimed. One way to handle this is to ensure your application is checkpointing as it processes data, i.e., that it’s writing to storage outside the VM itself, like Google Cloud Storage or a database. As an example, we have sample code for using a shutdown script to write a checkpoint file into a Cloud Storage bucket. For web applications behind a load balancer, consider using the 30-second preemption notice to drain connections to that VM so the traffic can be shifted to another VM. Some customers also choose to automate the shutdown of preemptible VMs on a rolling basis before the 24-hour period is over, to avoid having multiple VMs shut down at the same time if they were launched together. 5. Try autoscaling Another great way to save on costs is to run only as much capacity as you need, when you need it. As we mentioned earlier, typically around 70% of capacity is needed for steady-state usage, but when you need extra capacity, it’s critical to have it available. In an on-prem environment, you need to purchase that extra capacity ahead of time. In the cloud, you can leverage autoscaling to automatically flex to increased capacity only when you need it. Compute Engine managed instance groups are what give you this autoscaling capability in Google Cloud. You can scale up gracefully to handle an increase in traffic, and then automatically scale down again when the need for instances is lowered (downscaling). You can scale based on CPU utilization, HTTP load balancing capacity, or Stackdriver Monitoring metrics. This gives you the flexibility to scale based on what matters most to your application. High costs do not computeAs we’ve shown above, there are many ways to optimize your Compute Engine costs. Monitoring your environment and understanding your usage patterns is key to understanding the best options to start with, taking the time to model your baseline costs up front. Then, there are a wide variety of strategies to implement depending on your workload and current operating model. For more on cost management, check out our cost management video playlist. And for more tips and tricks on saving money on other GCP services, check out our blog posts on Cloud Storage, Networking and BigQuery cost optimization strategies. We have additional blog posts coming soon, so stay tuned!
Quelle: Google Cloud Platform

Introducing more maintenance controls for Cloud SQL

Routine maintenance is a part of every database user experience—it’s how we ensure you get the performance improvements and new feature updates that keep your business running smoothly and securely. But we get it: Nobody likes downtime, no matter how brief. That’s why we’re pleased to announce that Cloud SQL, our fully managed database service for MySQL, PostgreSQL, and SQL Server, now lets you have more control over when your instances undergo routine maintenance. This includes two top-requested features: advanced notification and maintenance rescheduling.Understanding Cloud SQL maintenanceBefore describing these new controls, let’s answer a few questions we often hear about the maintenance that Cloud SQL performs.What is maintenance?To keep your databases stable and secure, Cloud SQL automatically patches and updates your database instance (MySQL, Postgres, and SQL Server), including the underlying operating system. To perform maintenance, Cloud SQL must temporarily take your instances offline.What is a maintenance window?Cloud SQL offers maintenance windows to minimize the impact of planned maintenance downtime to your applications and your business. Maintenance windows allow you to control when maintenance occurs.Cloud SQL’s maintenance windows are entirely optional. They are applied per instance, which means you can choose to enforce maintenance windows on some of your instances, but not on others. We hear that users often find maintenance windows most valuable for production instances and less valuable, or even unneeded, for test and development instances. With Cloud SQL, you’re in control.Are maintenance windows used for anything else?Cloud SQL also uses maintenance windows to deliver new functionality, including performance improvements, that requires us to temporarily take your instances offline. When new functionality and performance improvements are released, they are documented in our release notes.When does maintenance occur?The maintenance window you set defines the hour and day when an update occurs. Define your preferred maintenance window so that those updates will only happen when database activity is low, for example, on Saturday at midnight. Additionally, you can specify the order of update for your instance relative to other instances in the same project (“Earlier” or “Later”). Earlier timing is useful for test instances, allowing you to see the effects of an update the week before it reaches your production instances. What are the new controls?You can now receive notifications one week in advance of incoming maintenance activities, helping you prepare for upcoming maintenance. If needed, you can choose to reschedule maintenance after being notified. For example, you can delay maintenance up to one week, or you can apply it immediately.Getting started with Cloud SQL’s new maintenance controlsYou’ll start by setting up maintenance notifications in your project.If you haven’t already, set a maintenance window for your instance: On the Cloud SQL Instance details page, click Edit maintenance preferences, as shown here:Next, opt in to notifications: Set the Cloud SQL Maintenance Window option in the Cloud Console Communications page, and select ON under Email. When notifications are enabled, you’ll get an email seven days in advance of the maintenance event.Now, you can view upcoming maintenance for all of your instances at a glance. On the Instances page, you can add a column for maintenance. When an instance is scheduled for maintenance, the date is listed in the maintenance column.If needed, reschedule maintenance. You can choose to reschedule maintenance after being notified.For more, find a detailed overview of maintenance and setup steps in our online documentation.What’s next for Cloud SQLSupport for additional maintenance controls has been a top request from users and its launch is an important milestone for Cloud SQL. You can look forward to additional notification types, like machine-readable notifications, in the future. Have more ideas? Let us know what other features and capabilities you need with our Issue Tracker and by joining the Cloud SQL discussion group. We’re glad you’re along for the ride, and we look forward to your feedback!
Quelle: Google Cloud Platform

File storage made easier with NetApp Cloud Volumes, now GA

At both NetApp and Google Cloud, we share a mission to offer our users a top-notch file service in Google Cloud. Whether you’re moving workloads to cloud or deploying net-new applications in cloud that need file interfaces, our aim is to offer a highly available, feature-rich, and high-performing file service. We recently announced two major milestones toward our mission: the general availability of Cloud Volumes Service for Google Cloud, and the availability of this service in the Google Cloud region in London. Cloud Volumes Service for Google Cloud is a fully managed service that lets you run enterprise apps in the cloud without compromising performance or flexibility. This has helped businesses move high-performance file workloads to cloud without refactoring apps, and unlock the power of their data with tightly integrated data solutions that are optimized and validated for Google Cloud. We’ve heard how useful this service is for enterprises to add flexibility for users. “We’ve been searching for a suitable replacement for CephFS and to do away with the headaches of maintaining our own internal Ceph stack,” said Aleem Shah, senior operations engineer, Prowler.io. “We tried many alternatives, with NetApp Cloud Volumes Service coming out on top in terms of its ever-expanding feature set, reliability, and pricing. The service support provided by NetApp was unmatched in all categories.”Here are some details on new service features included in these announcements.A new regionWith the added support for Cloud Volumes Service in the UK, the service is now available in five regions: three in the U.S., one in the U.K., and one in Germany.An availability SLA of 99.9% monthly uptimeA key requirement of running enterprise applications on cloud is ensuring reliability and availability of the infrastructure services. Cloud Volumes Service now offers a monthly uptime guarantee of 99.9% for your cloud volumes within a given region. This is the culmination of foundational hardening and scale testing accomplished by the joint teams of NetApp and Google SRE. The uptime guarantee is backed by a service-level agreement (SLA). You can confidently deploy production and business-critical workloads on Cloud Volumes Service. Support through Google Cloud SupportTo streamline customer experience and reduce time to resolution, you can now initiate support cases directly through Google Cloud Support, just like you do for any other Google Cloud service. With the general availability, we now have the cross-organization support framework with systems, telemetry, and enablement components in place. This means we can offer more efficient triaging and timely resolution of issues—essential for resolving problems in business-critical applications. Volume latency metrics in Stackdriver For enterprise workloads that support business processes, web content, batch runs, and so on, storage latency has significant implications on user experience, application efficacy, and job runtimes. Access to the right storage monitoring metrics can make the debugging process a lot easier for issues related to user experience or job runtimes. In addition to IOPS and storage throughput per cloud volume, you can now view both read and write latency metrics (in milliseconds) per cloud volume in Stackdriver. Latency metrics can help solve issues and understand the performance profile of the storage infrastructure for longer term planning. Find more details in the Monitoring Cloud Volumes section of the documentation.IAM controls in the Cloud ConsoleSecuring access to services within projects is a requirement for general enterprise and security-sensitive workloads. In addition to using gcloud commands, you can now assign the predefined cloud volume roles of admin and viewer to users, service accounts, and groups directly in Cloud Console, in the IAM section. For details about these roles, check out the Permissions for Cloud Volumes Service section in the documentation. Here’s what that looks like:We’re excited to bring this general availability and new features to help you get better outcomes with your production applications and workloads running on Cloud Volumes Service. Log in to Cloud Console and discover Cloud Volumes Service for yourself!
Quelle: Google Cloud Platform