April 2022 - Seite 31 von 43 - Cloud Computing Köln

Cloud Bigtable is a natural destination for Apache HBase workloads, as it is a fully managed service that is compatible with the HBase API. As a result, many customers running business-critical applications with large-scale data and low-latency needs consider migrating to Bigtable.However, migrating from HBase to Bigtable can still be challenging since you typically have to pause your applications for migration downtime. In addition, some companies choose to write custom tools, which require extensive resources to build and test, adding months to the migration process.Today, we’re announcing that Live Migrations from Apache HBase to Cloud Bigtable are now generally available. This enables faster and simpler migrations from HBase to Bigtable to ensure accurate data migration, reduce migration effort, and provide a better overall developer experience.HBase to Bigtable migrations just got easier Historically, you would need to manually create tables in Bigtable from your existing HBase tables and execute several steps to export and import data, define target tables, and validate data integrity. This process can be tedious, especially if the migration requires moving multiple tables or pre-splitting tables. At Google Cloud, we’re always trying to find ways to make migrations from HBase to Bigtable even easier for our customers. Our latest Live Migration features aim to provide a more straightforward, more efficient, and proven way to migrate data from HBase to Bigtable with minimal downtime. All together, they provide the necessary components to complete a seamless live migration.We have built four new features:Schema Translation Tool automates table schema conversions.HBase Bigtable Replication Library minimizes downtime for live migrations.Snapshot Import Tool easily imports HBase snapshots into Cloud Bigtable. Migration Validation Tool ensures accurate data migration.Now, you can automate the migration process and facilitate end-to-end data pipelines. The Schema Translation Tool fully automates table conversion by connecting to HBase, copying the table schema, and creating similar tables in Bigtable. You can also import HBase snapshots and validate data migration for a more seamless migration process with our Snapshot Import and Migration Validation tools. The HBase Bigtable Replication Library, which becomes available today, removes the need for building custom migration tools. It allows you to use HBase replication to sequence bulk imports and live writes correctly, ensuring consistent performance during migration of large workloads.How live migrations from HBase to Bigtable worksHBase provides asynchronous replication between clusters for various use cases like disaster recovery and data aggregation workloads. The HBase Bigtable Replication Library enables Bigtable to be added as an HBase cluster replication target. HBase to Bigtable replication enables customers to sync mutations happening on their HBase cluster to Bigtable, providing near-zero downtime migrations from HBase to Cloud Bigtable. The following diagram shows a live replication from HBase to Bigtable:The HBase Cluster is the source database, which can be located in an on-premises network, another cloud provider, or managed data services. Once enabled, live replication allows all the writes happening on the source cluster to be replicated to the target Bigtable Instance.Before enabling replication, you will need to create all the tables from HBase with the same column families in Bigtable. You can use the Schema Translation Tool to create target tables in Bigtable based on your existing HBase schema. To enable replication, the source cluster must be able to connect to the target Bigtable instance.Get started with HBase to Bigtable live migrationsTo learn more about HBase to Bigtable Live Migrations and how to get started, please visit our documentation page.To learn more about Bigtable:Create an instanceor try it out with a Bigtable Qwiklab.Check out these Youtube video tutorials for a step-by-step introduction to how Bigtable can be used for real-world applications like Personalization and Fraud detection.Related ArticleMigrating table schemas from Apache HBase to Cloud BigtableCloud Bigtable Schema Translation Tool: A new tool to seamlessly create new tables in Cloud Bigtable from an existing Apache HBase table’…Read Article
Quelle: Google Cloud Platform

8. April 2022

da Agency

Database observability for developers: announcing Cloud SQL Insights for MySQL (in preview)

In 2021, we announced Cloud SQL Insights, an easy to use tool that helps developers quickly understand and resolve database performance issues in Cloud SQL for PostgreSQL. We designed Insights using open standards, with intuitive visualizations and modern architectures, such as microservices, in mind.Insights became one of the fastest adopted new capabilities in Cloud SQL, so it was no surprise that many of our customers asked for the same functionality for MySQL, the other most widely used open-source relational database. Today, we’re happy to announce that support for Cloud SQL for MySQL is available in preview.Let’s take the opportunity to review the benefits of Cloud SQL Insights for application developers and DBAs and the significance of MySQL support.Why is it difficult to keep up with database performance problems?Application development teams are shipping features faster than ever before. With the rise of DevOps, what was once an event a few times a year has shifted to multiple releases per week, or even several a day. In addition, more application users are spread around the globe, continuously creating load on your applications. If performance issues aren’t properly identified during development, they may show up in production. When performance issues arise, the database is often the root cause. More ownership of the development lifecycle has now moved to software developers. However, developers may not have the skills or tools to solve database performance problems, and they’ll need to wait their turn if they turn to database experts, such as DBAs, for help. DBAs are now a scarce resource, heavily outnumbered by developers and overloaded as they try to meet the demand of today’s data-driven businesses. More often than not, urgent performance issues are left unresolved, resulting in poor end user experiences.Introducing Cloud SQL Insights for MySQLInsights, a feature of Cloud SQL on Google Cloud, helps developers diagnose and provide a resolution for MySQL database performance problems on their own, allowing for faster iteration on application development and freeing DBAs to work on more strategic tasks like data modeling, query optimization, and data analysis.Let’s dive a little deeper into Insights’ capabilities. Application-centric database observabilityTraditional monitoring tools provide a query-centric view of performance. This is a limitation since it creates a disconnect between performance and application code, especially with modern architectures. Insights provides developers with database monitoring through the lens of the application. You can use tags to associate queries with specific business functions, such as payments, inventory, business analytics, and shipping. For example, you can quickly evaluate the database load contributed by specific microservices or user flows.Insights provides a holistic view of performance organized by business function rather than query. Here’s a look at database load sorted by tags:In many applications, Object-Relational Mappers (ORMs) simplify database query development. However, ORMs tend to generate inefficient queries that are very difficult to diagnose. Insights integrates with Sqlcommenter, an open-source library that enables your ORMs to add comments to SQL statements to help you identify which application code is causing problems.Sqlcommenter automatically creates query tags, so you don’t need to make any changes to application code. It supports many popular ORMs, such as Hibernate, Spring, Django, Flask, and others. Learn more about Sqlcommenter.Self-service experience for query diagnosticsDatabase performance troubleshooting brings a few important challenges: quickly determining which query is causing the problem, finding the root cause of the problem, and identifying the specific application code causing the problem. Today, you likely have to rely on multiple tools to correlate data, a task that requires expertise and time. In the cloud, the challenge increases further as development teams often use multiple database engines for different use cases.Insights for MySQL allows you to move from detection to diagnosis seamlessly, using a single interface. Similar to Cloud SQL Insights for PostgreSQL, you can identify query performance problems early with pre-built dashboards. Here’s an example:This end-to-end application trace helps determine the source of the problematic query in context, including by model, view, controller, route, user, and host. And visual query plans give insights into the root cause of performance problems. Here’s a look at that view:Insights manages the collection of telemetry for diagnostics and reduces the performance impact on the database and the time you spend managing third-party monitoring products. To safeguard your data, all performance metrics are protected by Google Cloud’s enterprise-class security, privacy, and compliance.Database observability with your favorite tools and open standardsFor DevOps to work effectively, it’s imperative that the database be included seamlessly in the software development lifecycle, allowing a variety of stakeholders like developers, SREs, platform engineers, and DBAs to collaborate on troubleshooting database performance issues. This requires access to database telemetry across various enterprise monitoring tools—for example, developers want access to database traces in their favorite APM tool, while SREs want to access critical database signals in their operational dashboard.Insights helps increase database observability within existing tools, enabling developers and operations teams to address issues early and save time on troubleshooting. Unlike alternative approaches that require installing APM agents on top of the database, which can cause security concerns and performance overhead, Insights provides database metrics and traces through the open standard OpenTelemetry and the Cloud Monitoring and Cloud Trace APIs. As a result, it’s easy to execute end-to-end tracing in your existing tools and get a full-stack view of all your environments, from the application to the database. Insights also integrates with Cloud Monitoring, letting you quickly create custom dashboards and alerts on query metrics or tags and receive notifications via email, SMS, Slack, PagerDuty, and more. Cloud Monitoring also allows you to build customized dashboards.Sign up for the Insights for MySQL previewCloud SQL Insights for MySQL is now in preview. Sign up to join, and be sure to provide your feedback!Related ArticleDatabase observability for developers: introducing Cloud SQL InsightsNew Insights tool helps developers quickly understand and resolve database performance issues on Cloud SQL.Read Article
Quelle: Google Cloud Platform