da Agency

Unlocking the hidden value of data: Launching Dataflow Templates for Elastic Cloud

A key strategic priority for many customers that we work with is to unlock the value of the data they already have. Up to 73%of all data within an enterprise goes unused for analytics. “Dark data” is a commonly used term to refer to this and industry observers believe that most enterprises are only leveraging a small percentage of their data for analytics. While there are many dimensions to unlocking data for analytics, one fundamental component is the need to make data available across systems so that users are not limited by organization and technological silos. To that end, we launched Dataflow Templates to address the challenge of making data seamlessly available across various systems that a typical enterprise needs to deal with. Dataflow Templates, built on the rich, scalable and fault tolerant data processing capabilities of Dataflow and Apache Beam, provides a turnkey, utility-like experience for common data ingestion and replication tasks.  Dataflow Templates are prepackaged data ingestion and replication pipelines for many of the common needs that users have to make data available to all their users and systems so that they can make the most of the data. We provide a number of Dataflow Templates including some of the most popular ones such as Pub/Sub to BigQuery, Apache Kafka to BigQuery, Cloud Spanner to Cloud Storage and Data Masking/Tokenization (using Cloud DLP). We continue to make more Templates available and recently launched Templates for Datastream to create up-to-date replicated tables of your data in Database into BigQuery for analytics.Today, we announce a new set of Templates  we  developed in partnership with Elastic.  These Templates allow data engineers, developers, site reliability engineers (SREs) and security analysts to ingest data from BigQuery, Pub/Sub and Cloud Storage to the Elastic Stack with just a few clicks in the Google Cloud Console. Once created, Dataflow Templates run in a serverless fashion: there is no infrastructure to size and setup, no servers to manage and more importantly, no distributed system expertise is required to setup and operate these Templates. Once the data is in Elastic, users can easily search and visualize their data with Elasticsearch and Kibana to complement the analytics they are able to perform on Google Cloud.“It’s never been easier for our customers to explore and analyze their data with Elastic in Google Cloud. By leveraging Google Dataflow templates, customers can ingest data from Google Cloud services, such as BigQuery, Cloud Storage and Pub/Sub without having to install and manage agents,” said Uri Cohen, Product Lead,Elastic Cloud. Getting started with these Templates is very easy, first head to Dataflow in the Google Cloud Console, select Create Job From Template and then select one of the Elastic Templates from the dropdown.The team at Elastic has also published a series of blog posts with detailed information on using these Templates. Ingest data directly from Google Pub/Sub into Elastic using Google DataflowIngest data directly from Google BigQuery into Elastic using Google DataflowIngest data directly from Google Cloud Storage into Elastic using Google DataflowAdditionally, you can  find more information, including other Templates available at Getting started with Dataflow Templates.
Quelle: Google Cloud Platform

da Agency

Liberating your mainframe data with Confluent and Google Cloud

Are you looking for the best way to migrate and replicate your mainframe data? Google Cloud and Confluent have teamed up to provide an end-to-end solution for connecting your mainframe application data with the advanced analytics capabilities of Google Cloud.In this article, we will discuss how you can use Confluent Connect to replicate messages from IBM MQ and Db2 to Google Cloud. This allows you to work with your mainframe data in the cloud, and enables you to build new applications and analytical capabilities using Google Cloud’s machine learning solutions. You also benefit by reducing impact on your production mainframe workloads, and reducing general purpose compute costs. In other words, you can continue using your mainframe to run your mission-critical business workloads while setting your data in motion for innovation.Here’s an example use case that demonstrates how using the Confluent MQ connector with Google Cloud can impact your bottom line. One of our customers is saving millions of dollars per year on mainframe cycles by leveraging z Integrated Information Processor (zIIP) engines for data processing.Moving these workloads to zIIP, off of GP (general purpose) compute, and away from CHINIT (Channel Initiator) routes directly leads to reduced MSU licensing. As an example, a customer in the financial services industry saw a 50% reduction in CPU usage per message. These cost savings can enable you to direct budget resources toward differentiating activities, such as commercializing your valuable mainframe data to open up new revenue streams and improve customer service.On the technical side, Confluent guarantees exactly-once message semantics, preserives message order and unleashes that data to be accessed by existing and new applications that need a high throughput, low latency event driven architecture. This means that you can rely on the accuracy and consistency of your data in Google Cloud as if you were querying it directly from your mainframe database.Once you have this data in your Confluent cluster, you can leverage  the combined capabilities of Confluent and Google Cloud. You can modernize the way your consumers access your data by providing a single, standard source of truth without impacting production services. Confluent integrates directly with Apigee, Google Cloud’s API platform for developing and managing APIs.Because Confluent integrates with BigQuery, you can also leverage the advanced analytical capabilities of BigQuery ML and Vertex AI to realize value from your latent mainframe data, and build new systems of insight that were not possible on the mainframe. And most of all, you can open up new avenues for innovation by allowing consumers to access the data when they need it, speeding up time to value and enabling faster business decisions.You now have a bridge to cloud for your mainframe application data. Get started by deploying Confluent from the Google Cloud marketplace.Related ArticleBeyond mainframe modernization: The art of possibilitiesMainframe modernization has been a hot topic over the past decade or so. Over time, the term “modernization” itself is manifested in many…Read Article
Quelle: Google Cloud Platform