Export Google Cloud data into Elastic Stack with Dataflow templates

At Google Cloud, we’re focused on solving customer problems while supporting a thriving partner ecosystem. Many of you use third-party monitoring solutions to keep a tab on your multi-cloud or hybrid cloud environments, be it for IT operations, security operations, application performance monitoring, or cost analysis. At the same time, you’re looking for a cloud-native way to reliably export your Google Cloud logs, events, and alerts at scale.As part of our efforts to expand the set of purpose-built Dataflow templates for these common data movement operations, we launched three Dataflow templates to export Google Cloud data into your Elastic Cloud or your self-managed Elasticsearch deployment: Pub/Sub to Elasticsearch (streaming), Cloud Storage to Elasticsearch (batch) and BigQuery to Elasticsearch (batch).In this blog post, we’ll show you how to set up a streaming pipeline to export your Google Cloud logs to Elastic Cloud using the Pub/Sub to Elasticsearch Dataflow template. Using this Dataflow template, you can forward to Elasticsearch any message that can be delivered to a Pub/Sub topic, including logs from Cloud Logging or events such as security findings from Cloud Security Command Center. The step-by-step walkthrough covers the entire setup, from configuring the originating log sinks in Cloud Logging, to setting up Elastic integration with GCP in Kibana UI, to visualizing GCP audit logs in a Kibana dashboard.Push vs. PullTraditionally, Elasticsearch users have the option to pull logs from Pub/Sub topics into Elasticsearch via Logstash or Beats as a data collector. This documented solution works well, but it does include tradeoffs that need to be taken into account:Requires managing one or more data collectors with added operational complexity for high availability and scale-out with increased log volumeRequires external resource access to Google Cloud by giving permissions to aforementioned data collectors to establish subscription and pull data from one or more Pub/Sub topics.We’ve heard from you that you need a more cloud-native approach that streams logs directly into your Elasticsearch deployment without the need to manage an intermediary fleet of data collectors. This is where the managed Cloud Dataflow service comes into play: A Dataflow job can automatically pull logs from a Pub/Sub topic, parse payloads and extract fields, apply an optional JavaScript user-defined function (UDF) to transform or redact the logs, then finally forward to the Elasticsearch cluster.Set up logging export to ElasticsearchThis is how the end-to-end logging export looks:Below are the steps that we’ll walk through:Set up Pub/Sub topics and subscriptionsSet up a log sinkSet IAM policy for Pub/Sub topicInstall Elastic GCP integrationCreate API key for ElasticsearchDeploy Pub/Sub to the Elastic Dataflow templateView and analyze GCP logs in KibanaSet up Pub/Sub topics and subscriptionsFirst, set up a Pub/Sub topic that will receive your exported logs, and a Pub/Sub subscription that the Dataflow job can later pull logs from. You can do so via the Cloud Console or via CLI using gcloud. For example, using gcloud looks like this:Note: It is important to create the subscription before setting up the Cloud Logging sink to avoid losing any data added to the topic prior to the subscription getting created.Repeat the same steps for the Pub/Sub deadletter topic that holds any undeliverable message, due to pipeline misconfigurations (e.g. wrong API key) or inability to connect to Elasticsearch cluster:Set up a Cloud Logging sinkCreate a log sink with the previously created Pub/Sub topic as destination. Again, you can do so via the Logs Viewer, or via CLI using gcloud logging. For example, to capture all logs in your current Google Cloud project (replace [MY_PROJECT]), use this code:Note: To export logs from all projects or folders in your Google Cloud organization, refer to aggregated exports for examples of “gcloud logging sink” commands. For example, provided you have the right permissions, you may choose to export Cloud Audit Logs from all projects into one Pub/Sub topic to be later forwarded to Elasticsearch.The output of this last command is similar to this:Take note of the service account [LOG_SINK_SERVICE_ACCOUNT] returned. It typically ends with @gcp-sa-logging.iam.gserviceaccount.com.Set IAM policy for Pub/Sub topicFor the sink export to work, you need to grant the returned sink service account a Cloud IAM role so it has permission to publish logs to the Pub/Sub topic:If you created the log sink using the Cloud Console, it will automatically grant the new service account permission to write to its export destinations, provided you own the destination. In this case, it’s Pub/Sub topic my-logs.Install Elastic GCP integrationFrom Kibana web UI, navigate to ‘Integrations’ and search for GCP. Select ‘Google Cloud Platform (GCP)’ integration, then click on ‘Add Google Cloud Platform (GCP)’.In the following screen, make sure to uncheck ‘Collect Google Cloud Platform (GCP) … (input: gcp-pubsub)’ since we will not rely on pollers to pull data from Pub/Sub topic, and rather on Dataflow pipeline to stream that data in.Create API key for ElasticsearchIf you don’t already have an API key for Elasticsearch, navigate to ‘Stack Management’ > ‘API keys’ to create an API key from Kibana web UI. Refer to Elastic docs for more details on Elasticsearch API keys. Take note of the base64-encoded API key which will be used later by your Dataflow pipeline to authenticate with Elasticsearch.Before proceeding, take also note of your Cloud ID which can be found from Elastic Cloud UI under ‘Cloud’ > ‘ Deployments’.Deploy Pub/Sub to Elastic Dataflow pipelineThe Pub/Sub to Elastic pipeline can be executed either from the Console, gcloud CLI, or via a REST API call (more detail here). Using the Console as example, navigate to the Dataflow Jobs page, click ‘Create Job from Template’ then select “Cloud Pub/Sub to Elasticsearch” template from the dropdown menu. After filling out all required parameters, the form should look similar to this:Click on ‘Show Optional Parameters’ to expand the list of optional parameters.Enter ‘audit’ for ‘The type of logs…’ parameter to specify the type of dataset we’re sending in order to populate the corresponding GCP audit dashboard available in the GCP integration you enabled previously in Kibana:Once you click “Run job”, the pipeline will start streaming events to Elastic Cloud after a few minutes. You can visually check correct operation by clicking on the Dataflow job and selecting the “Job Graph” tab, which should look as below. In our test project, the Dataflow step WriteToElasticsearch is sending a little over 2,800 elements per second at that point in time:Now head over to Kibana UI, and navigate under ‘Observability’ > ‘Overview’  to quickly inspect that your GCP audit logs are being ingested in Elasticsearch:Visualize GCP Audit logs in KibanaYou can now view Google Cloud audit logs from your Kibana UI search interface. Navigate to either ‘Observability’ > ‘Logs’ > ‘Stream’ or ‘Analytics’ > ‘ Discover’, and type the following simple query in KQL to filter for GCP audit logs only:data_stream.dataset:”gcp.audit”The above table was produced after selecting the following fields as columns in order to highlight who did what to which resource:protoPayload.authenticationInfo.principalEmail – WhoprotoPayload.methodName – WhatprotoPayload.serviceName – Which (service)protoPayload.resourceName – Which (resource)Open GCP Audit dashboard in KibanaNavigate to ‘Analytics’ > ‘Dashboards’, and search for ‘GCP’. Select ‘[Logs GCP] Audit’ dashboard to visualize your GCP audit logs. Among other things, this dashboard displays a map view of where your cloud activity is coming from, a timechart of activity volume, and a breakdown of top actions and resources acted on.But wait, there’s more!Pub/Sub to Elasticsearch Dataflow template is meant to abstract away the heavy-lifting when it comes to reliably collecting voluminous logs in near real-time. At the same time, it offers advanced customizations to tune the pipeline to your own requirements with optional parameters such as delivery batch size (in number of messages or bytes) for throughput, retry settings (in number of attempts or duration) for fault tolerance, and a custom user-defined function (UDF) to transform the output messages before delivery to Elasticsearch. To learn more about Dataflow UDFs along with specific examples, see Extend your Dataflow templates with UDFs.In addition to Pub/Sub to Elasticsearch Dataflow template, there are two new Dataflow templates to export to Elasticsearch depending on your use case: Cloud Storage to Elasticsearch: Use this Dataflow template to export rows from CSV files in Cloud Storage into Elasticsearch as JSON documents.BigQuery to Elasticsearch: Use this Dataflow template to export rows from a BigQuery table (or results from a SQL query) into Elasticsearch. This is particularly handy to forward billing data by Cloud Billing or assets metadata snapshots by Cloud Asset Inventory, both of which can be natively exported to BigQuery. What’s next?Refer to our user docs for the latest reference material on all Google-provided Dataflow templates including the Elastic Dataflow ones described above. We’d like to hear your feedback and feature requests. You can create an issue directly in the corresponding GitHub repo, or create a support case directly from your Cloud Console, or ask questions in our Stack Overflow forum.To get started with Elastic Cloud on Google Cloud, you can subscribe via Google Cloud Marketplace and start creating your own Elasticsearch cluster on Google Cloud within minutes. Refer to Elastic getting started guide for step by step instructions.AcknowledgementsWe’d like to thank several contributors within and outside Google for making these Elastic Dataflow templates available for our joint customers:Prathap Kumar Parvathareddy, Strategic Cloud Engineer, GoogleAdam Quan, Solutions Architect, ElasticMichael Yang, Product Manager, ElasticSuyog Rao, Engineering Manager, ElasticRelated ArticleExtend your Dataflow template with UDFsLearn how to easily extend a Cloud Dataflow template with user-defined functions (UDFs) to transform messages in-flight, without modifyi…Read Article
Quelle: Google Cloud Platform

Bare Metal Solution launches its second Sydney location

We’re excited to announce the launch of a second zone in Sydney for Bare Metal Solution (BMS) in order to bring BMS to our customers around the world. With state of the art hardware certified for enterprise applications, BMS is the best place to run your Oracle workloads with ultra-low latency to Google Cloud. When customers migrate their workloads to BMS, they are delighted by its simplicity, its similarities to a typical on-prem environment, and its ability to run standard Oracle technologies like Oracle RAC, Data Guard, and Recovery Manager (RMAN). Whether BMS is your long-term Oracle solution, or you consider BMS as a stepping stone to exit Oracle and modernize to a managed database, e.g. Cloud Spanner or Cloud SQL, we’re continuing to invest in BMS to meet you where you are today. Just in 2021, we’ve launched BMS in Iowa, Seoul, and Zurich, bringing our total global presence to 13 regions. In addition to new BMS regions, we also announced El Carro, our newly open-sourced Kubernetes operator to automate your Oracle deployments and ongoing management. We open-sourced El Carro to share our Kubernetes best-practices with the industry for wherever you’re running Oracle: on BMS, on-prem, or even in another cloud. We’re excited to continue contributing to the open source community and developing El Carro to make managing Oracle databases a breeze. In addition to adding a second zone in Sydney for BMS, Google Cloud also recently launched our second Australian cloud region in Melbourne. Here’s how Lendlease plans to leverage both the new cloud region in Melbourne and our new Sydney location for BMS: “The new cloud region in Melbourne helps us to achieve our objectives around operating at a global scale, including BMS for mission-critical services. It will also provide benefits around business continuity acting as DR for our portfolio.” – Harvey Worton, Co-Group CIO, LendleaseIf BMS is your stepping stone to modernize from Oracle to a managed database service, then we hope you watched Google Cloud Next (you still can). During Next ‘21, we detailed how Google is making major investments and breakthroughs with our Cloud Databases. For example, our recent launch of a Spanner PostgreSQL interface has generated tremendous interest with our customers. This interface, along with recent granular instance sizing, has democratized Spanner, making it more accessible and cementing it as an unparalleled industry offering with 99.999% availability, starting at $65 per month, with a new PostgreSQL interface. Our other most popular Oracle BMS migration target, Cloud SQL, is blazing new trails with industry-leading observability for developers with Cloud SQL Insights. Cloud SQL Insights is available at no additional cost and provides a single interface with powerful tools for database monitoring through the lens of the application. In whichever way you want to use BMS, whether as a long term home for Oracle or as a means to modernize your applications at your own pace, Google Cloud is here  to help you on your journey so you can focus on your core business and take advantage of all that Cloud has to offer, rather than running hardware or managing a data center.To learn more about BMS and how Google can help you in your journey to the cloud, please check out our 2021 Next session “Accelerating your move to the cloud with managed databases” (video, slides, whitepaper) or our BMS product page where you can contact sales.Related ArticleBare Metal Solution: new regions, new servers, and new certificationsCheck out new regions and a smaller 8-core server, plus HIPAA and PCI-DSS compliance, for Bare Metal Solution to move your Oracle workloa…Read Article
Quelle: Google Cloud Platform

What is Cloud CDN and how does it work?

No matter what your app or website does, chances are that your users are distributed across various locations and are not necessarily close to your servers. This means the requests travel long distances across the public internet , leading to inconsistent and sometimes frustrating user experiences. That’s where Cloud CDN comes in!Click to enlargeWhat is Cloud CDN?Cloud CDN is a content delivery network that accelerates your web and video content delivery by using Google’s global edge network to bring content as close to your users as possible. As a result latency, cost, and load on your backend servers is reduced, making it easier to scale to millions of users. Global anycast IP provides a single IP for global reach. It enables Google Cloud to route users to the nearest edge cache automatically and avoid DNS propagation delays that can impact availability. It supports HTTP/2 end-to-end and the QUIC protocol from client to cache. QUIC is a multiplexed stream transport over UDP, which reduces latency and makes it ideal for lossy mobile networks.How does Cloud CDN work?Let’s consider an example to understand how Cloud CDN works:When a user makes a request to your website or app, the request is routed to the closest Google edge node (we have over 120 of these!) for fast and reliable traffic flow. From there the request gets routed to the global HTTPS Load Balancer to the backend or origin.With Cloud CDN enabled, the content gets directly served from the cache — a group of servers that store and manage cacheable content so that future requests for that content can be served faster.The cached content is a copy of cacheable web assets (JavaScript, CSS), images, video, and other content that is stored on your origin servers.Cloud CDN automatically caches this content when you use the recommended “cache mode” to cache all static content. If you need more control, you can direct Cloud CDN by setting HTTP headers on your responses. You can also force all content to be cached; just know that this ignores  the “private”, “no-store”, or “no-cache” directives in Cache-Control response headers.When the request is received by Cloud CDN it looks for the cached content using a cache key. This is typically the URI, but you can customize the cache key to remove protocol, hosts,or query strings.If a cached response is found in the Cloud CDN cache, the response is retrieved from the cache and sent to the user. This is called a cache hit. When a cache hit occurs, Cloud CDN looks up the content by its cache key and responds directly to the user, shortening the round-trip time and reducing the load on the origin server.The first time that a piece of content is requested, Cloud CDN can’t fulfill the request from the cache because it does not have it in cache. This is called a cache miss. When a cache miss occurs, Cloud CDN might attempt to get the content from a nearby cache. If the nearby cache has the content, it sends it to the first cache by using cache-to-cache fill. Otherwise, it just sends the request to the origin server. The maximum lifetime of the object in a cache is defined by the TTLs, or time to live values, set by the cache directives for each HTTP response or cache mode. When the TTL expires, the content is evicted from cache.How to use Cloud CDN You can set up Cloud CDN through gCloud CLI, Cloud Console, or the APIs. Since Cloud CDN uses Cloud Load Balancing to provide routing, health checking, and anycast IP support, it can be enabled by easily selecting a checkbox while setting up your backends or origins. Cloud CDN makes it easy to serve web and media content using Google Cloud Storage. You just upload your content to a Cloud Storage bucket, set up your load balancer, and enable caching. To enable hybrid architectures spanning across clouds and on-premises, Cloud CDN and HTTP(S) Load Balancing also support external backends.SecurityData is encrypted at rest and in transit from Cloud Load Balancing to the backend for end-to-end encryption.You can programmatically sign URLs and cookies to limit video segment access to authorized users only. The signature is validated at the CDN edge and unauthorized requests are blocked right there! On a broader level, you can enable SSL for free using Google managed certs! For a more in-depth look into Cloud CDN check out the documentation. For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.Related ArticleTraffic Director explained!If your application is deployed in a microservices architecture then you are likely familiar with the networking challenges that come wit…Read Article
Quelle: Google Cloud Platform

TechEd 2021: What’s new for SAP customers on Google Cloud

SAP’s TechEd 2021 is the premier developer and IT-focused virtual event for any business that runs — or is planning to run — SAP applications, and it gets under way this week. What better time to take stock and share how, together, we’re creating value for SAP customers? For Google Cloud, 2021 has easily been the most productive year in our working relationship with SAP, as we continue our joint focus on helping enterprises complete their journey to the cloud as quickly and successfully as possible. In the last 12 months alone, we’ve added three times the number of SAP customers as the year prior. Here’s what new and what’s just ahead for SAP customers on Google Cloud: Google Cloud rollouts for SAP Business Technology PlatformWe’re excited to announce that a new instance of SAP Business Technology Platform (BTP) is now available on Google Cloud, and we’re kicking off our BTP rollouts in North America and Europe during Q4 of 2021. This will be followed by BTP rollouts for our customers in Asia Pacific Japan (APJ) with a BTP instance available in India during Q1 of 2022.SAP Business Technology Platform enables SAP customers to move quickly to leverage the cloud, even for relatively complex use cases. And it puts an emphasis on freedom of choice by allowing customers to choose the right blend of cloud solutions to address their unique needs — and to pivot quickly with new capabilities as those needs change.Perhaps most important, running BTP (or any other set of SAP technologies) on Google Cloud gives customers the benefits of operating on the industry’s most extensive, fully owned and operated, global network infrastructure. That gives SAP customers some unique and very important network security and latency advantages.HANA Cloud to launch on Google CloudSAP’s fully managed HANA Cloud is an important component of SAP’s cloud portfolio, offering a database that can serve OLTP, OLAP, and xOLTP use cases. Like any business-critical application running in the cloud, it’s important to ensure that HANA Cloud delivers flawless performance — something that can be a major challenge for SAP customers looking to minimize network latency.That makes Google Cloud, with its low-latency global networking capabilities and flexible Compute Engine application platform, an ideal partner for SAP customers running HANA Cloud as part of their SAP portfolio. HANA Cloud also provides data virtualization capabilities with BigQuery, allowing customers flexibility in managing their enterprise data footprint. We’re pleased to announce that HANA Cloud is officially launched on Google Cloud in North America, with expansion to EMEA customers expected later this quarter (Q4 2021). We’re also launching HANA Cloud on Google Cloud for our APAC customers, with a rollout in India planned for early 2022.Even SAP customers who aren’t currently Google Cloud customers can take advantage of our HANA Cloud rollout — giving them the same performance and latency benefits even if they choose to run other SAP applications on other hyperscalers or within on-premises environments.Filestore Enterprise for RISE with SAPAnother key element of our SAP partnership centers on RISE with SAP — a “business transformation as a service” offering that gives enterprises a faster, lower-risk strategy for migrating their SAP environments to the cloud. Now, SAP is integrating our Filestore Enterprise technology as a standard part of RISE with SAP S/4HANA Cloud, private edition solutions. With its 99.99% regional-availability SLA, Filestore Enterprise is a fully-managed, cloud-native NFS solution designed for applications that demand high availability. With a few clicks (or a few lines of gCloud CLI or API calls), you can simply provision NFS shares that are synchronously replicated across three zones within a region. Filestore also lets you take periodic snapshots of the file system, and easily recover an individual file or an entire file system in less than 10 minutes from any prior snapshot recovery points.For HEC/RISE with SAP customers, Filestore Enterprise will be an important capability in terms of ensuring high availability, high performance, high levels of redundancy, and higher SLAs for business-critical SAP applications. It’s another way to give SAP customers a simpler, lower-risk path to the cloud.VPC Network Peering: More value for RISE with SAPFinally, we wanted to call out an existing Google Cloud capability with significant benefits for RISE with SAP customers: VPC Network Peering.When SAP implements a PCE or HEC environment on Google Cloud, it also manages the VPC — a virtualized network that provides the same functionality, performance, and security benefits as a dedicated physical network — for that environment. When you connect your SAP environment with Google Cloud services and applications through VPC Network Peering, the traffic between peered VPC networks remains securely within Google’s network. This is a big advantage for customers concerned with protecting business-critical SAP applications and data while connecting seamlessly to other applications running on Google Cloud.A partnership built to create lasting valueAll of these announcements build on the things that already make Google Cloud a natural partner for SAP customers: security, reliability, flexibility, and freedom of choice. Best of all, as a trusted partner in this journey with SAP and its customers, we know that even bigger successes are coming on the road ahead. Learn more about Google Cloud solutions for SAP customers.
Quelle: Google Cloud Platform