da Agency

7 trends to watch in the API economy

Organizations across every industry are aiming to achieve digital transformation initiatives to stay competitive, which increasingly include overcoming the complexity of hybrid and multicloud environments. With resources stretched and margins getting thinner, application programming interfaces (APIs) and the API economy will continue to play a critical role in connecting services, applications, and clouds.So, what do you need to know to help your organization continue to thrive? Here are the biggest trends shaping the future of the API economy. 1. API security takes center stage.As the number of APIs has surged to support the digital ecosystems and applications of the world, so has concern over security risks. The primary attack surface is no longer the application—Gartner® predicts that in API abuses will become the most-frequent attack vector responsible for data breaches during 2022.1 Today, securing APIs requires visibility across all application interactions and observing, analyzing, and taking action at every level of the technology stack. The fixed security perimeters of the past are slowly vanishing as enterprises become more open and distributed. We expect to see more organizations moving away from network-focused security towards models that prioritize zero-trust and layered defense based on encryption, application identity, and strong authentication and authorization.2. Microservices APIs are gaining speed. Since the launch of Kubernetes (K8s) in 2014, we’ve witnessed a massive industry shift towards decomposing systems into microservices. But enabling thousands of microservices to work together runs the risk of introducing technology sprawl and even recreating many of the pitfalls of monolithic architecture. Even when enterprises adopt concepts like domain-driven design, they often end up with “microservice islands” that support a single application. That’s why we expect to see microservices APIs becoming a new focus for IT departments in 2022 and beyond. For instance, Kubernetes-native ingress gateways are emerging as the crucial point of technology optionality for application modernization. Ingress gateways sit in front of Kubernetes resources, allowing you to deliver new business value by exposing those services with APIs. Whether these APIs power internal applications or developer and partner applications, ingress gateways are key enablers for abstraction, discovery, and easy consumption of underlying resources. This helps maintain the internal agility of microservices while also providing stability—an API contract—for developers and their applications. 3. Event-driven architecture (EDA) continues its big comeback.If we told you five years ago that event-driven architecture (EDA) would still be hot in 2022, we probably would have been laughed out of the room. Yet, according to a 2021 survey from Solace, the majority of organizations (85%) recognize the critical business value in adopting EDA. This seemingly old development concept has found a new life in many interactions, such as serverless, asynchronous, and streaming use cases. In particular, event-driven architecture is becoming the preferred paradigm to support API-agnostic, real-time data exchange between microservices. Still, EDA technology falls short of many of today’s digital requirements, so we expect to see more and more solutions appear with security, access control, and governance capabilities. While event-driven APIs are increasing and driving business innovation, they do not guarantee success. Like any approach, failures can often outnumber successes if teams scale project development faster than operations can keep up. Many organizations have some technology to implement event-driven design but lack the vision of event thinking to conceive, design, and manage event-driven interactions. Enterprises will need to adopt a new way of thinking and undergo proper preparation and skills assessments to ensure initiatives succeed. 4. GraphQL will accelerate BFF (Backends for Frontends). While REST remains the most commonly used standard for designing APIs, GraphQL has been gaining popularity with developers for its flexibility and ease of use. Gartner predicts that by 2025, more than 50% of enterprises will use GraphQL in production, up from less than 10% in 2021.2 One of GraphQL’s standout benefits is that it enables developers to seamlessly query data from multiple apps and services with a single API call. This is particularly useful for the Backends-for-Frontends (BFF) pattern, since it allows companies to aggregate and deliver the exact data requested by a client from multiple microservices without overfetching data or the need to package an API and endpoint for each specific client type. In 2022, we’ll see GraphQL adoption continue to increase and accelerate the use of the BFF pattern.5. It’s not one API gateway to rule them all anymore. APIs are now the “crown jewels” of modern software development, providing the connective tissue that powers nearly every digital product available today. With the majority of software still living on premises, hybrid API architectures enable organizations to continue to innovate while bridging to existing technologies.But heterogeneous distributed IT environments also mean multiple clouds and software vendors for building and deploying APIs, which is particularly challenging given the pervasiveness of hybrid deployments. As a result, expect to see more “multi-APIM by design” during 2022 to enable hybrid API management that is lightweight, portable, and scalable. API management will need to ensure that every API in an organization, regardless of where it comes from, reaps all the benefits of API management—including consistent visibility, governance and security guardrails, and analytics. 6. Conversational APIs go mainstream. We live in the age of voice experiences, from smart homes to vehicle infotainment systems. These interactive experiences aren’t replacing traditional IVR technology altogether, but they dominate for specific devices like smart speakers, smartwatches, and in-car infotainment. There’s a pressing need to enable voice experiences as quickly as other more traditional interfaces, such as mobile apps and websites. As a result, many chat or voice apps and systems will join the API economy and start exposing their own conversational APIs to generate new interactions and innovation across many different platforms. 7. From Shadow IT to Strategic ITAPIs have gained a reputation as the new Shadow IT, as developers often build them without alerting their IT and security teams. Increasingly, more IT departments will start to recognize that APIs are the key way to expose data from tools and apps for internal use. APIs will play a critical role in governing data access, encouraging shadow API publishers to follow a standard process. Instead of an imminent threat, shadow IT can become a technology advantage that will produce new internal apps and drive application innovation that can be leveraged across the entire organization. Learn more about how Apigee is continuing to innovate and helping companies stay ahead of the top API trends.GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.1. Gartner Webinar, API Security: Protect your APIs from Attacks and Data BreachesMark O’Neill, Dionisio Zumerle, Jul 15 20212. Gartner, Predicts 2022: APIs Demand Improved Security and Management, Shameen Pillai, Jeremy D’Hoinne, John Santoro, Mark O’Neill, Sham Gill, 6 December 2021Related ArticleFirst dedicated AI management and MLOps platform on Google Cloud MarketplaceBeing able to make strategic decisions based on automated predictions and more targeted data saved the company $1.3M Euros in two fiscal …Read Article
Quelle: Google Cloud Platform

da Agency

Google Cloud Partner Consumption Packs accelerate customer journeys to the cloud

When we launched Partner Advantage, we committed to making it predictable and easy for partners to drive business with us. Since launch, those commitments have been validated by channel experts like CRN, which gave Partner Advantage a 5-Star award for 2022, and by the fact that our partners have seen impressive growth* across virtually every facet of their business. I am pleased to announce that our commitments endure today with the launch of Consumption Packs for Deal Acceleration Funds and Partner Services Funds (DAF and PSF for partners).  Inspired by partner feedback, these new packages are designed to accelerate all stages of a customer’s journey to the cloud, and make it even easier and faster for partners to do business with us. Consumption Packs are purpose-built so that partners can plan and initiate customer projects much more quickly, with predictable funding. They include assets and templates that allow partners to deliver Google Cloud designed and validated infrastructure, application migration and modernization plans to customers faster than ever–particularly for customers beginning their journey to the cloud. Based on learnings gathered from thousands of customer deployments, these turnkey packs have been designed by our partners and Google Cloud Partner Engineering and Professional Services’ teams. Here’s a brief look at Consumption Packs in action:Consumption Packs offer pre-approved, curated templates and assets to simplify and shorten the process for most common projects.For Deal Acceleration Funds (DAF), packages include everything partners need to conduct assessments, workshops and proofs-of-concept so they can quickly meet customers where they are on their journey to the cloud.For Partner Services Funds (PSF), packages are structured so that partners can develop cloud ready foundation and migration plans that align with Google Cloud priority solution areas. Packages have pre-determined funding levels to enable faster deployments.  Partners still have the option to engage with Google Cloud Partner Advantage and their customers through customized requests, as they always have. This is ideally suited for projects that require a tailored approach to meet unique customer requirements.We are launching nine consumption packs today focused on key enterprise workloads, with a vision toward introducing additional packages to cover more solutions. Partners can explore Consumption Packs now by visiting the Partner Advantage portal. We welcome your continued feedback and suggestions, and look forward to helping our customers achieve new levels of growth and success, together.See you in the cloud.* The Google Cloud Business Opportunity For Partners, a commissioned Total Economic Impact™ study conducted by Forrester Consulting, October 2021Related ArticleGoogle Cloud Partners driving Retail and Commerce InnovationLearn how Google Cloud Partner Advantage partners help customers solve real-world business challenges in retail and ecommerce.Read Article
Quelle: Google Cloud Platform

da Agency

BigQuery Omni innovations enhance customer experience to combine data with cross cloud analytics

IT leaders pick different clouds for many reasons, but the rest of the company shouldn’t be left to navigate the complexity of those decisions.  For data analysts, that complexity is most immediately felt when navigating between data silos. Google Cloud has invested deeply in helping customers break down these barriers inherent in a disparate data stack.  Back in October 2021, we launched BigQuery Omni to help data analysts access and query data across the barriers of multi cloud environments. We are continuing to double down in cross-cloud analytics: a seamless approach to view, combine, and analyze data across-clouds with a single pane of glass.  Earlier this year, one of BigQuery Omni’s early adopters, L’Oreal, discussed the merits of a cross-cloud analytics to maximize their data platform.  We know that enterprises need to analyze data without needing to move or copy any data.  We also know that enterprises sometimes need to move small amounts of data between clouds to leverage unique cloud capabilities.  A full cross-cloud analytics solution offers the best of both worlds: analyzing data where it is and flexibility to replicate data when necessary. Last week, we launched BigQuery Omni cross-cloud transfer to help customers with combining data across clouds.  From a single-pane-of-glass, data analysts, scientists, and engineers, can load data from AWS and Azure to BigQuery without any data pipelines. Because it is all managed in SQL, it is accessible among all levels of an organization.  We have designed this feature to provide three core benefits:Usability: With one single-pane-of-glass, users tell BigQuery to filter and move data between clouds without any context-switchingSecurity: With a federated identity model, users don’t have to share or store credentials between cloud providers to access and copy their dataLatency: With data movement managed by BigQuery’s high-performance storage API, users can effortlessly move just the relevant data without having to wait for complex pipesA core use case that we have heard from customers is to combine point of sales (PoS) data from AWS/Azure with Google Analytics data and create a consolidated purchase prediction model. Here’s a demo of that:As you saw in the demo, a data analyst can drive end-to-end workflows across clouds. They can transform data using BigQuery Omni, they can load data using cross-cloud transfer, and they can train an ML model all in SQL. This empowers them to drive real business impact by providing the ability to:  Improve training data by de-deuplicating users across datasetsImprove accuracy of marketing segmentation modelsImprove Return on Ads Spend and save potentially millions for enterprise campaignsBut we’re not stopping there, we will continue to build upon this experience by providing more BigQuery native tools for our customers to assist with smart data movement.  Over time, our cross-cloud data movement will be built on pipeless pipelines:  A cross-cloud lakehouse without the fuss. Get involved with the preview and start participating in our development process by submitting this short form.Related ArticleBigQuery Omni now available for AWS and Azure, for cross cloud data analyticsBigQuery Omni helps teams break down silos by securely and cost-effectively analyzing data across clouds.Read Article
Quelle: Google Cloud Platform

da Agency

Google Cloud launches Optimization AI: Cloud Fleet Routing API to help customers make route planning easier

Increased customer demands, climate concerns, and higher delivery expectations from the consumer market are just a few of the emerging macro trends that retail, logistics, and consumer packaged goods industries face. To stay competitive in this environment, businesses need to improve execution, both to serve customers and to address the onslaught of logistical and environmental factors that go into delivery. Significant and pressing among those factors: companies need to make intelligent use of new technologies that not only accelerate business impact, but also contribute to sustainability improvements required in a post-carbon digital economy.That’s why today, Google Cloud is announcing the public general availability ofOptimization AI and of its first feature: Cloud Fleet Routing (CFR) API. Optimization AI helps businesses plan for the future, decide the appropriate action, and clarify the business impact; CFR specifically helps vehicle fleet operators to create accurate fleet-wide routing plans at scale.Quickly create complex plans reliablyOur customers have shared with us that they would like to send out the fleet as early as possible, allow for later cutoff times, and easily incorporate changes to plans. In addition, providing an effective action plan that customers can quickly and reliably disseminate to their fleet is a must. By intelligently harnessing the scale of Google Cloud’s infrastructure, CFR can solve small route planning problems in seconds and offers batch processing that can scale up to model even very large  logistics operations.Example illustration of routes built by Cloud Fleet Routing APIFor example, to continue improving its delivery service and leverage one of its main strategic pillars, which is to offer the fastest delivery in the country, Brazilian retailer Magalu adopted the CFR API to optimize last-mile route for same-day delivery, all while navigating a dense logistics network that involves more than 2.1 million shipments per month. “Google’s Cloud Fleet Routing API not only helps us meet our growing demand with efficient routes, it also gives us more reliability. The Google Cloud solution allows you to calculate multiple pickups and deliveries on the same route in seconds. Thus, we improve our ability to plan routes quickly, flexibly and accurately, generating more productivity and scalability. This also contributes to more efficient costs, which keeps us more competitive,” said Talita Paschoini, IT Director at Magalu. Reduce CO2 and deliver business impactIt is not easy to accurately predict and fulfill competitive ETAs while making cost-effective allocation and scheduling decisions, and it is near-impossible to do so with manual planning. With our CFR API, customers have options to bring their own distance matrix or use the native integration with Google Maps to take advantage of AI-infused ETA. Sharing accurate visit times and infeasibilities as early as possible helps customers build trust with the brand. With CFR, it has never been easier to spot opportunities to reduce CO2 footprints via fuel savings, promote work-life balance by providing optimal workloads, and meet customers’ demands.“At Instabox we have environmental sustainability close to heart – and optimizing our routes is a crucial way of bringing more value to customers while limiting our environmental footprint,” said Staffan Gabrielsson, CTO and co-founder at Instabox. “Google’s Cloud Fleet Routing API improved the efficiency of our routes and reduced the planning time. Our customers are getting their deliveries faster through our increased capacity and we are reducing operating costs at the same time.”Navigate through uncertainty with easeLast mile operations are fraught with disruptions: change of delivery time, vehicle malfunction, last-minute workforce changes, and the list goes on. To help our customers stay agile, the CFR API lets them reoptimize their existing plan up to 20 times a day without incurring any additional cost. This ability is especially useful for the last mile, where dynamic changes such as traffic congestion or ad hoc requests can require frequent re-optimization. As a result,  customers can use our CFR API to plan effective disruption responses worry-free, so that operators can quickly take the best action with consumers as soon as possible.Get startedTo learn more about how to get started on the Optimization AI: Cloud Fleet Routing API, check out the product page. For an integrated last mile planning and execution use case, check out Last Mile Fleet Solution to get  started. Please join us for the Spotlight session “Shining a light on supply chain and logistics” to learn more about how Google Cloud can support demand shaping, inventory positioning, and perfect fulfillment by providing the end-to-end visibility you need for optimized planning and decision making.We are thrilled to partner with you to help deliver joy to your customersFor additional support, select partners have created discovery sessions, proof of value projects, and pilots to help you get started.Related ArticleIntroducing Last Mile Fleet Solution: Maximize fleet performance from ecommerce order to doorstep deliveryStarting today, Last Mile Fleet Solution is available to help fleet operators create exceptional delivery experiences, from start to finish.Read Article
Quelle: Google Cloud Platform

da Agency

How to migrate from Apache HBase to Cloud Bigtable with Live Migrations

Cloud Bigtable is a natural destination for Apache HBase workloads, as it is a fully managed service that is compatible with the HBase API. As a result, many customers running business-critical applications with large-scale data and low-latency needs consider migrating to Bigtable.However, migrating from HBase to Bigtable can still be challenging since you typically have to pause your applications for migration downtime. In addition, some companies choose to write custom tools, which require extensive resources to build and test, adding months to the migration process.Today, we’re announcing that Live Migrations from Apache HBase to Cloud Bigtable are now generally available. This enables faster and simpler migrations from HBase to Bigtable to ensure accurate data migration, reduce migration effort, and provide a better overall developer experience.HBase to Bigtable migrations just got easier Historically, you would need to manually create tables in Bigtable from your existing HBase tables and execute several steps to export and import data, define target tables, and validate data integrity. This process can be tedious, especially if the migration requires moving multiple tables or pre-splitting tables. At Google Cloud, we’re always trying to find ways to make migrations from HBase to Bigtable even easier for our customers. Our latest Live Migration features aim to provide a more straightforward, more efficient, and proven way to migrate data from HBase to Bigtable with minimal downtime. All together, they provide the necessary components to complete a seamless live migration.We have built four new features:Schema Translation Tool automates table schema conversions.HBase Bigtable Replication Library minimizes downtime for live migrations.Snapshot Import Tool easily imports HBase snapshots into Cloud Bigtable. Migration Validation Tool ensures accurate data migration.Now, you can automate the migration process and facilitate end-to-end data pipelines. The Schema Translation Tool fully automates table conversion by connecting to HBase, copying the table schema, and creating similar tables in Bigtable. You can also import HBase snapshots and validate data migration for a more seamless migration process with our Snapshot Import and Migration Validation tools. The HBase Bigtable Replication Library, which becomes available today, removes the need for building custom migration tools. It allows you to use HBase replication to sequence bulk imports and live writes correctly, ensuring consistent performance during migration of large workloads.How live migrations from HBase to Bigtable worksHBase provides asynchronous replication between clusters for various use cases like disaster recovery and data aggregation workloads. The HBase Bigtable Replication Library enables Bigtable to be added as an HBase cluster replication target. HBase to Bigtable replication enables customers to sync mutations happening on their HBase cluster to Bigtable, providing near-zero downtime migrations from HBase to Cloud Bigtable. The following diagram shows a live replication from HBase to Bigtable:The HBase Cluster is the source database, which can be located in an on-premises network, another cloud provider, or managed data services. Once enabled, live replication allows all the writes happening on the source cluster to be replicated to the target Bigtable Instance.Before enabling replication, you will need to create all the tables from HBase with the same column families in Bigtable. You can use the Schema Translation Tool to create target tables in Bigtable based on your existing HBase schema. To enable replication, the source cluster must be able to connect to the target Bigtable instance.Get started with HBase to Bigtable live migrationsTo learn more about HBase to Bigtable Live Migrations and how to get started, please visit our documentation page.To learn more about Bigtable:Create an instanceor try it out with a Bigtable Qwiklab.Check out these Youtube video tutorials for a step-by-step introduction to how Bigtable can be used for real-world applications like Personalization and Fraud detection.Related ArticleMigrating table schemas from Apache HBase to Cloud BigtableCloud Bigtable Schema Translation Tool: A new tool to seamlessly create new tables in Cloud Bigtable from an existing Apache HBase table’…Read Article
Quelle: Google Cloud Platform

da Agency

Database observability for developers: announcing Cloud SQL Insights for MySQL (in preview)

In 2021, we announced Cloud SQL Insights, an easy to use tool that helps developers quickly understand and resolve database performance issues in Cloud SQL for PostgreSQL. We designed Insights using open standards, with intuitive visualizations and modern architectures, such as microservices, in mind.Insights became one of the fastest adopted new capabilities in Cloud SQL, so it was no surprise that many of our customers asked for the same functionality for MySQL, the other most widely used open-source relational database. Today, we’re happy to announce that support for Cloud SQL for MySQL is available in preview.Let’s take the opportunity to review the benefits of Cloud SQL Insights for application developers and DBAs and the significance of MySQL support.Why is it difficult to keep up with database performance problems?Application development teams are shipping features faster than ever before. With the rise of DevOps, what was once an event a few times a year has shifted to multiple releases per week, or even several a day. In addition, more application users are spread around the globe, continuously creating load on your applications. If performance issues aren’t properly identified during development, they may show up in production. When performance issues arise, the database is often the root cause. More ownership of the development lifecycle has now moved to software developers. However, developers may not have the skills or tools to solve database performance problems, and they’ll need to wait their turn if they turn to database experts, such as DBAs, for help. DBAs are now a scarce resource, heavily outnumbered by developers and overloaded as they try to meet the demand of today’s data-driven businesses. More often than not, urgent performance issues are left unresolved, resulting in poor end user experiences.Introducing Cloud SQL Insights for MySQLInsights, a feature of Cloud SQL on Google Cloud, helps developers diagnose and provide a resolution for MySQL database performance problems on their own, allowing for faster iteration on application development and freeing DBAs to work on more strategic tasks like data modeling, query optimization, and data analysis.Let’s dive a little deeper into Insights’ capabilities. Application-centric database observabilityTraditional monitoring tools provide a query-centric view of performance. This is a limitation since it creates a disconnect between performance and application code, especially with modern architectures. Insights provides developers with database monitoring through the lens of the application. You can use tags to associate queries with specific business functions, such as payments, inventory, business analytics, and shipping. For example, you can quickly evaluate the database load contributed by specific microservices or user flows.Insights provides a holistic view of performance organized by business function rather than query. Here’s a look at database load sorted by tags:In many applications, Object-Relational Mappers (ORMs) simplify database query development. However, ORMs tend to generate inefficient queries that are very difficult to diagnose. Insights integrates with Sqlcommenter, an open-source library that enables your ORMs to add comments to SQL statements to help you identify which application code is causing problems.Sqlcommenter automatically creates query tags, so you don’t need to make any changes to application code. It supports many popular ORMs, such as Hibernate, Spring, Django, Flask, and others. Learn more about Sqlcommenter.Self-service experience for query diagnosticsDatabase performance troubleshooting brings a few important challenges: quickly determining which query is causing the problem, finding the root cause of the problem, and identifying the specific application code causing the problem. Today, you likely have to rely on multiple tools to correlate data, a task that requires expertise and time. In the cloud, the challenge increases further as development teams often use multiple database engines for different use cases.Insights for MySQL allows you to move from detection to diagnosis seamlessly, using a single interface. Similar to Cloud SQL Insights for PostgreSQL, you can identify query performance problems early with pre-built dashboards. Here’s an example:This end-to-end application trace helps determine the source of the problematic query in context, including by model, view, controller, route, user, and host. And visual query plans give insights into the root cause of performance problems. Here’s a look at that view:Insights manages the collection of telemetry for diagnostics and reduces the performance impact on the database and the time you spend managing third-party monitoring products. To safeguard your data, all performance metrics are protected by Google Cloud’s enterprise-class security, privacy, and compliance.Database observability with your favorite tools and open standardsFor DevOps to work effectively, it’s imperative that the database be included seamlessly in the software development lifecycle, allowing a variety of stakeholders like developers, SREs, platform engineers, and DBAs to collaborate on troubleshooting database performance issues. This requires access to database telemetry across various enterprise monitoring tools—for example, developers want access to database traces in their favorite APM tool, while SREs want to access critical database signals in their operational dashboard.Insights helps increase database observability within existing tools, enabling developers and operations teams to address issues early and save time on troubleshooting. Unlike alternative approaches that require installing APM agents on top of the database, which can cause security concerns and performance overhead, Insights provides database metrics and traces through the open standard OpenTelemetry and the Cloud Monitoring and Cloud Trace APIs. As a result, it’s easy to execute end-to-end tracing in your existing tools and get a full-stack view of all your environments, from the application to the database. Insights also integrates with Cloud Monitoring, letting you quickly create custom dashboards and alerts on query metrics or tags and receive notifications via email, SMS, Slack, PagerDuty, and more. Cloud Monitoring also allows you to build customized dashboards.Sign up for the Insights for MySQL previewCloud SQL Insights for MySQL is now in preview. Sign up to join, and be sure to provide your feedback!Related ArticleDatabase observability for developers: introducing Cloud SQL InsightsNew Insights tool helps developers quickly understand and resolve database performance issues on Cloud SQL.Read Article
Quelle: Google Cloud Platform

da Agency

Boost the power of your transactional data with Cloud Spanner change streams

Data is one of the most valuable assets in today’s digital economy. One way to unlock the value of your data is to give it life after it’s first collected. A transactional database, like Cloud Spanner, captures incremental changes to your data in real time, at scale, so you can leverage it in more powerful ways. Cloud Spanner is our fully managed relational database that offers near unlimited scale, strong consistency, and industry-leading high availability of up to 99.999%. The traditional way for downstream systems to use incremental data that’s been captured in a transactional database is through change data capture (CDC), which allows you to trigger behavior based on changes to your database, such as a deleted account or an updated inventory count.Today, we are announcing Spanner change streams, coming soon, that lets you capture change data from  Spanner databases and easily integrate it with other systems to unlock new value. Change streams for Spanner goes above and beyond the traditional CDC capabilities of tracking inserts, updates, and deletes. Change streams are highly flexible and configurable, letting you track changes on exact tables and columns or across an entire database. You can replicate changes from Spanner to BigQuery for real-time analytics, trigger downstream application behavior using Pub/Sub, and store changes in Google Cloud Storage (GCS) for compliance. This ensures you have the freshest data to optimize business outcomes. Change streams provides a wide range of options to integrate change data with other Google Cloud services and partner applications through turnkey connectors, including custom Dataflow processing pipelines or the change streams read API.Spanner consistently processes over 1.2 billion requests per second. Since change streams are built right into Spanner, you not only get industry-leading availability and global scale—you also don’t have to spin up any additional resources. The same IAM permissions that already protect your Spanner databases can be used to access change streams queries.Change stream queries are protected by spanner.databases.select, and change stream DDL operations are protected by spanner.databases.updateDdl.Change streams in actionIn this section, we’ll look at how to set up a change stream that sends change data from Spanner to an analytic data warehouse in BigQuery.Creating a change stream As discussed above, a change stream tracks changes on an entire database, a set of tables, or a set of columns in a database. Each change stream can have a retention period of anywhere from one day to seven days, and you can set up multiple change streams to track exactly what you need for your specific business objectives. First, we’ll create a change stream on a table called InventoryLedger. This table tracks inventory changes on two columns: InventoryLedgerProductSku and InventoryLedgerChangedUnits with a 7-day retention period.Change recordsEach change record contains a wealth of information, including primary key, the commit timestamp, transaction ID, and of course, the old and new values of the changed data, wherever applicable. This makes it easy to process change records as an entire transaction, in sequence based on their commit timestamp, or individually as they arrive, depending on your business needs. Back to the inventory example, now that we’ve created a change stream on the InventoryLedger table, all inserts, updates, and deletes on this table will be published to the InventoryStream change stream. These changes are strongly consistent with the commits on the InventoryLedger table: When a transaction commit succeeds, the relevant changes will automatically persist in the change stream. You never have to worry about missing a change record.Processing a change streamThere are numerous ways that you can process change streams depending on the use case:Analytics: You can send the change records to BigQuery, either as a set of change logs or by updating the tables.  Event triggering: You can send change logs to Pub/Sub for further processing by downstream systems. Compliance: You can retain the change log to Google Cloud Storage for archiving purposes. The easiest way to process change stream data is to use our Spanner connector for Dataflow, where you can take advantage of Dataflow’s built-in pipelines to BigQuery, Pub/Sub, and Google Cloud Storage. The diagram below shows a Dataflow pipeline that processes this change stream and imports change data directly into BigQuery.Alternatively, you can build a custom Dataflow pipeline to process change data with Apache Beam. In this case, we provide a Dataflow connector that outputs change data as an Apache Beam PCollection of DataChangeRecord objects. For even more flexibility, you can use the underlying change streams query API. The query API is a powerful interface that lets you read directly from a change stream to implement your own connector and stream changes to the pipeline of your choice. On the query API side, a change stream is divided into multiple partitions, which can be used to query a change stream in parallel for higher throughput. Spanner dynamically creates these partitions based on load and size. Partitions are associated with a Spanner database split, allowing change streams to scale as effortlessly as the rest of Spanner.Get started with change streamsWith change streams, your Spanner data follows you wherever you need it, whether that’s for analytics with BigQuery, for triggering events in downstream applications, or for compliance and archiving. Change streams are highly flexible and configurable —allowing you to capture change data for the exact data you care about, and for the exact period of time that matters for your business. And because change streams are built into  Spanner, there’s no software to install, and you get external consistency, high scale, and up to 99.999% availability.There’s no extra charge for using change streams, and you’ll pay only for extra compute and storage of the change data at the regular Spanner rates.To get started with Spanner, create an instance, or try it out with a Spanner Qwiklab.We’re excited to see how Spanner change streams will help you unlock more value out of your data!Related ArticleCloud Spanner myths bustedThe blog talks about the 7 most common myths and elaborates the truth for each of the myths.Read Article
Quelle: Google Cloud Platform

da Agency

Meet Google’s unified data and AI offering

Without AI, you’re not getting the most out of your data.Without data, you risk stale, out-of-date, suboptimal models.But most companies are still struggling with how to keep these highly interdependent technologies in sync and operationalize AI to take meaningful action from data.We’ve learned from Google’s years of experience in AI development how to make data-to-AI workflows as cohesive as possible and as a result our data cloud is the most complete and unified data and AI solution provider in the market. By bridging data and AI, data analysts can take advantage of user-friendly, accessible ML tools, and data scientists can get the most out of their organization’s data. All of this comes together with built-in MLOps to ensure all AI work — across teams — is ready for production use. In this blog we’ll show you how all of this works, including exciting announcements from the Data Cloud Summit:Vertex AI Workbench is now GA bringing together Google Cloud’s data and ML systems into a single interface so that teams have a common toolset across data analytics, data science, and machine learning. With native integrations across BigQuery, Spark, Dataproc, and Dataplex data scientists can build, train and deploy ML models 5X faster than traditional notebooks. Introducing Vertex AI Model Registry, a central repository to manage and govern the lifecycle of your ML models. Designed to work with any type of model and deployment target, including BigQuery ML, Vertex AI Model Registry makes it easy to manage and deploy models. Use ML to get the most out of your data, no matter the formatAnalyzing structured data in a data warehouse, like using SQL in BigQuery, is the bread and butter for many data analysts. Once you have data in a database, you can see trends, generate reports, and get a better sense of your business. Unfortunately, a lot of useful business data isn’t in the tidy tabular format of rows and columns. It’s often spread out over multiple locations and in different formats, frequently as so-called “unstructured data” — images, videos, audio transcripts, PDFs — can be cumbersome and difficult to work with. Here, AI can help. ML models can be used to transcribe audio and videos, analyze language, and extract text from images—that is, to translate elements of unstructured data into a form that can be stored and queried in a database like BigQuery. Google Cloud’s Document AI platform, for example, uses ML to understand documents like forms and contracts. Below, you can see how this platform is able to intelligently extract structured text data from an unstructured document like a resume. Once this data is extracted, it can be stored in a data warehouse like BigQuery.Bring machine learning to data analysts via familiar toolsToday, one of the biggest barriers to ML is that the tools and frameworks needed to do ML are new and unfamiliar. But this doesn’t have to be the case. BigQuery ML, for example, allows you to train sophisticated ML models at scale using SQL code, directly from within BigQuery. Bringing ML to your data warehouse alleviates the complexities of setting up additional infrastructure and writing model code. Anyone who can write SQL code can train a ML model quickly and easily.Easily access data with a unified notebook interfaceOne of the most popular ML interfaces today are notebooks: interactive environments that allow you to write code, visualize and pre-process data, train models, and a whole lot more. Data scientists often spend most of their day building models within notebook environments. It’s crucial, then, that notebook environments have access to all of the data that makes your organization run, including tools that make that data easy to work with. Vertex AI Workbench, now generally available, is the single development environment for the entire data science workflow. Integrations across Google Cloud’s data portfolio allow you to natively analyze your data without switching between services:Cloud Storage: access unstructured dataBigQuery: access data with SQL, take advantage of models trained with BigQuery MLDataproc: execute your notebook using your Dataproc cluster for controlSpark: transform and prepare data with autoscaling serverless SparkBelow, you’ll see how you can easily run a SQL query on BigQuery data with Vertex AI Workbench.But what happens after you’ve trained the model? How can both data analysts and data scientists make sure their models can be utilized by application developers and maintained over time?Go from prototyping to production with MLOpsWhile training accurate models is important, getting those models to be scalable, resilient, and accurate in production is its own art, known as MLOps. MLOps allow you to:Know what data your models are trained onMonitor models in productionMake training process repeatableServe and scale model predictionsA whole lot more! (See the “Practitioners Guide to MLOps” whitepaper for a full and detailed overview of MLOps)Built-in MLOps tools within Vertex AI’s unified platform remove the complexity of model maintenance. Practical tools can help with everything from training and hosting ML models, managing model metadata, governance, model monitoring, and running pipelines – all critical aspects of running ML in production and at scale. And now, we’re extending our capabilities to make MLOps accessible to anyone working with ML in your organization. Easy handoff to MLOps with Vertex AI Model RegistryToday, we’re announcing Vertex AI Model Registry, a central repository that allows you to register, organize, track, and version trained ML models and is designed to work with any type of model and deployment target, whether that’s through BigQuery, Vertex AI, AutoML, custom deployments on GCP or even out of the cloud.Vertex AI Model Registry is particularly beneficial for BigQuery ML. While BigQuery ML brings the powerful scalability of BigQuery for batch predictions, using a data warehouse engine for real-time predictions just isn’t practical. Furthermore, you might start to wonder how to orchestrate your ML workflows based in BigQuery. You can now discover and manage  BigQuery ML models and easily deploy those models to Vertex AI for real-time predictions and MLOps tools. End-to-End MLOps with pipelinesOne of the most popular approaches to MLOps is the concept of ML pipelines: where each distinct step in your ML workflow from data preparation to model training and deployment are automated for sharing and reliably reproducing. Vertex AI Pipelines is a serverless tool for orchestrating ML tasks using pre-built components or your own custom code. Now, you can easily process data and train models with BigQuery, BigQuery ML, and Dataproc directly within a pipeline. With this capability, you can combine familiar ML development within BigQuery and Dataproc into reproducible, resilient pipelines and orchestrate your ML workflows faster than ever.See an example of how this works with the new BigQuery and BigQuery ML components.Learn more about how to use BigQuery and BigQuery ML components with Vertex AI Pipelines.Learn more and get started We’re excited to share more about our unified data and AI offering today at the Data Cloud Summit. Please join us for the spotlight session on our “AI/ML strategy and product roadmap” or the “AI/ML notebooks ‘how to’ session.”And if you’re ready to get hands on with Vertex AI, check out these resources:Codelab: Training an AutoML model in Vertex AICodelab: Intro to Vertex AI WorkbenchVideo Series: AI Simplified: Vertex AIGitHub: Example NotebooksTraining: Vertex AI: Qwik StartRelated ArticleWhat is Vertex AI? Developer advocates share moreDeveloper Advocates Priyanka Vergadia and Sara Robinson explain how Vertex AI supports your entire ML workflow—from data management all t…Read Article
Quelle: Google Cloud Platform

da Agency

BigLake: unifying data lakes and data warehouses across clouds

The volume of valuable data that organizations have to manage and analyze is growing at an incredible rate. This data is increasingly distributed across many locations, including  data warehouses, data lakes, and NoSQL stores. As an organization’s data gets more complex and proliferates across disparate data environments, silos emerge, creating increased risk and cost, especially when that data needs to be moved. Our customers have made it clear; they need help. That’s why today, we’re excited to announce BigLake, a storage engine that allows you to unify data warehouses and lakes. BigLake gives teams the power to analyze data without worrying about the underlying storage format or system, and eliminates the need to duplicate or move data, reducing cost and inefficiencies. With BigLake, users gain fine-grained access controls, along with performance acceleration across BigQuery and multicloud data lakes on AWS and Azure. BigLake also makes that data uniformly accessible across Google Cloud and open source engines with consistent security. BigLake extends a decade of innovations with BigQuery to data lakes on multicloud storage, with open formats to ensure a unified, flexible, and cost-effective lakehouse architecture.BigLake architectureBigLake enables you to:Extend BigQuery to multicloud data lakes and open formats such as Parquet and ORC with fine-grained security controls, without needing to set up new infrastructure.Keep a single copy of data and enforce consistent access controls across analytics engines of your choice, including Google Cloud and open-source technologies such as Spark, Presto, Trino, and Tensorflow.Achieve unified governance and management at scale through seamless integration with Dataplex.Bol.com, an early customer using BigLake, has been accelerating analytical outcomes while keeping their costs low:“As a rapidly growing e-commerce company, we have seen rapid growth in data. BigLake allows us to unlock the value of data lakes by enabling access control on our views while providing a unified interface to our users and keeping data storage costs low. This in turn allows quicker analysis on our datasets by our users.”—Martin Cekodhima, Software Engineer, Bol.comExtend BigQuery to unify data warehouses and lakes with governance across multicloud environmentsBy creating BigLake tables, BigQuery customers can extend their workloads to data lakes built on Google Cloud Storage (GCS), Amazon S3, and Azure data lake storage Gen 2. BigLake tables are created using a cloud resource connection, which is a service identity wrapper that enables governance capabilities. This allows administrators to manage access control for these tables similar to BigQuery tables, and removes the need to provide object store access to end users. Data administrators can configure security at the table, row or column level on BigLake tables using policy tags. For BigLake tables defined over Google Cloud Storage, fine grained security is consistently enforced across Google Cloud and supported open-source engines using BigLake connectors. For BigLake tables defined on Amazon S3 and Azure data lake storage Gen 2, BigQuery Omni enables governed multicloud analytics by enforcing security controls. This enables you to manage a single copy of data that spans BigQuery and data lakes, and creates interoperability between data warehousing, data lake, and data science use cases.Open interface to work consistently across analytic runtimes spanning Google Cloud technologies and open source engines Customers running open source engines like Spark, Presto, Trino, and Tensorflow through Dataproc or self managed deployments can now enable fine-grained access control over data lakes, and accelerate the performance of their queries. This helps you build secure and governed data lakes, and eliminate the need to create multiple views to serve different user groups. This can be done by creating BigLake tables from a supported query engine like Spark DDL, and using Dataplex to configure access policies. These access policies are then enforced consistently across the query engines that access this data – greatly simplifying access control management. Achieve unified governance & management at scale through seamless integration with DataplexBigLake integrates with Dataplex to provide management-at-scale capabilities. Customers can logically organize data from BigQuery and GCS into lakes and zones that map to their data domains, and can centrally manage policies for governing that data. These policies are then uniformly enforced by Google Cloud and OSS query engines. Dataplex also makes management easier by automatically scanning Google Cloud storage to register BigLake table definitions in BigQuery, and makes them available via Dataproc Metastore. This helps end users discover these BigLake tables for exploration and querying using both OSS applications and BigQuery. Taken together, these capabilities enable you to run multiple analytic runtimes over data spanning lakes and warehouses in a governed manner. This breaks down data silos and significantly reduces the infrastructure management, helping you to advance your analytics stack and unlock new use cases.What’s next?If you would like to learn more about BigLake, please visit our website. Alternatively, get started with BigLake today by using this quickstart guide, or contact the Google Cloud sales team.Related ArticleLimitless Data. All Workloads. For EveryoneRead about the newest innovations in data cloud announced at Google Cloud’s Data Cloud Summit.Read Article
Quelle: Google Cloud Platform

da Agency

Bringing together the best of both sides of BI with Looker and Data Studio

In today’s data world, companies have to consider many scenarios in order to deliver insights to users in a way that makes sense to them. On one hand, they must deliver trusted, governed reporting to inform mission-critical business functions. But on the other hand, they must enable a broad population to answer questions on their own, in an agile and self-serve manner. Data-driven companies want to democratize access to relevant data and empower their business users to get answers quickly. The trade-off is governance; it can be hard to maintain a single shared source of truth, granular control of data access, and secure centralized storage. Speed and convenience compete with trust and security. Companies are searching for a BI solution that can be easily used by everyone in an organization (both technical and non-technical), while still producing real-time governed metrics.Today, a unified experience for both self-serve and governed BI gets one step closer, with our announcement of the first milestone in our integration journey between Looker and Data Studio. Users will now be able to access and import governed data from Looker within the Data Studio interface, and build visualizations and dashboards for further self-serve analysis.This union allows us to better deliver a complete, integrated Google Cloud BI experience for our users. Business users will feel more empowered than ever, while data leaders will be able to preserve the trust and security their organization needs.This combination of ​self-serve and governed BI together will help enterprises make better data-driven decisions. Looker and Data Studio, Better Together Looker is a modern enterprise platform for business intelligence and analytics, that helps organizations build data-rich experiences tailored to every part of their business. Data Studio is an easy to use self-serve BI solution enabling ad-hoc reporting, analysis, and data mashups across 500+ data sets. Looker and Data Studio serve complementary use cases. Bringing Looker and Data Studio together opens up exciting opportunities to combine the strengths of both products and a broad range of BI and analytic capabilities to help customers reimagine the way they work with data. How will the integration work?This first integration between these products allows users to connect to the Looker semantic model directly from Data Studio. Users can bring in the data they wish to analyze, connect it to other available data sources, and easily explore and build visualizations within Data Studio. The integration will follow three principles with respect to governance:Access to data is enforced by Looker’s security features, the same way it is when using the Looker user interface.Looker will continue to be the single access point for your data.Administrators will have full capabilities to manage Data Studio and Looker together.How are we integrating these products?This is the first step in our roadmap that will bring these two products closer together. Looker’s semantic model allows metrics to be centrally defined and broadly used, ensuring a single version of the truth across all of your data. This integration allows the Looker semantic model to be used within Data Studio reports, allowing people to use the same tool to create reports that rely on both ad-hoc and governed data. This brings together the best of both worlds – a governed data layer, and a self-serve solution that allows analysis of both governed and ungoverned data.With this announcement, the following use cases will be supported:Users can turn their Looker-governed data into informative, highly customizable dashboards and reports in Data Studio.Users can blend governed data from Looker with data available from over 500 data sources in Data Studio, to rapidly generate new insights.Users can analyze and rapidly prototype ungoverned data (from spreadsheets, csv files, or other cloud sources) within Data Studio.Users can collaborate in real-time to build dashboards with teammates or people outside the company. When will this integration be available to use?The Data Studio connector for Looker is currently in preview. If you are interested in trying it out, please fill out this form.Next StepsThis integration is the first of many in our effort to bring Looker and Data Studio closer together. Future releases will introduce additional features to create a more seamless user experience across these two products. We are very excited to roll out new capabilities in the coming months and will keep you updated on our future integrations of the two products.Related ArticleLimitless Data. All Workloads. For EveryoneRead about the newest innovations in data cloud announced at Google Cloud’s Data Cloud Summit.Read Article
Quelle: Google Cloud Platform