da Agency

All You Need To Know About GitOps

opensource.substack.com – Dubbed as the agility sweet pot, containers are revolutionizing modern application development. They allow development teams to roll out multiple deployments, which helps meet their organizational de…
Quelle: news.kubernauts.io

da Agency

Dataflow Under the Hood: understanding Dataflow techniques

Editor’s note: This is the second blog in a three-part series examining the internal Google history that led to Dataflow, how Dataflow works as a Google Cloud service, and how it compares and contrasts with other products in the marketplace. Check out part 1: Dataflow Under the Hood: the origin story.In the first post in this series, we explored the genesis of Dataflow within Google, and talked about how it compares to Lambda Architectures. Now, let’s look a little closer at some of the key systems that power Dataflow. As mentioned in the first post, we’ve taken advantage of a number of technologies that we had built for previous systems, and also developed some new techniques.The origins of our timer system go back to the original MillWheel system, which provided users with direct access to setting timers for triggering processing logic. Conceptually, these are similar to other scheduling systems, where users set an arbitrary number of alarms, and the system is responsible for triggering those alarms at an appropriate time. For durability, we journal these timers to a backing store (like the Cloud Bigtabledatabase), and cache a subset of them in memory, such that all upcoming timers are in memory, and the cache can be refreshed asynchronously without putting storage reads on the hot path. One subtlety for timers arises from the need to support event time timers, which depend on the completeness of data for previous stages in order to trigger. We call these completion markers watermarks, and they are managed by a separate component, which communicates with all the nodes that are responsible for processing a given stage in order to determine current watermark value. This watermark component then publishes these values to all relevant downstream computations, which can use the watermark to trigger event time timers. To help illustrate why this is important, let’s consider a classic IoT use case—a manufacturing line where the equipment is instrumented with sensors. These sensors will emit reams of data, and the watermarks associated with the data will help group together this data by time, or perhaps by manufacturing run, and ensure we don’t miss data in our analysis just because it came in late or out of order.Understanding state managementState management in Dataflow takes advantage of a number of similar concepts as timers. State is journaled to a durable store, and cached for speed and efficiency. One thing that we learned from our experience with MillWheel was the need to provide useful abstractions for users to interact with state—some applications want to read and write the entirety of the stored state for each incoming record, but others want to read only a subset, or append to a list that is only occasionally accessed in full. In Dataflow, we’ve worked hard to provide relevant state abstractions that are integrated with the right caching and persistence strategies, so that the system is efficient and fast out of the box. We’ve also found it important to commit state modifications in an atomic operation with record processing. Many other systems take the approach of telling users to use an external state system, which is very difficult to get working correctly. Thinking back to the IoT use case we just discussed, Dataflow’s state management features would make it easy—meaning involving trivial amounts of user code—to do things like aggregating and counting equipment revolutions per minute, calculating the average temperature from a sensor over a given period of time, or determining the average deviation from a cutting or molding process without complicated retry logic for interacting with a secondary system.A major reason for the popularity of the Lambda Architecture is the challenges of providing exactly once processing in streaming processing systems (see this blog series for additional details). Dataflow provides exactly once processing for records by storing a fingerprint of each record that enters a given stage, and uses that to deduplicate any retries of that record. Of course, a naive strategy for this would create an unbounded number of fingerprints to check, so we use the watermark aggregator to determine when we can garbage-collect the fingerprints of records that have fully traversed the system. This system also makes ample use of caching, as well as some additional optimizations, including the use of rotating Bloom filters.One final aspect of Dataflow that we’ll touch upon is its ability to support autoscaling of pipeline resources. It is able to support dynamic scaling (both up and down) of both streaming and batch pipelines by having a means of dynamically reallocating the underlying work assignments that power the system. In the case of streaming pipelines, this corresponds to a set of key ranges for each computation stage, which can be dynamically shifted, split, and merged between workers to balance out the load. The system responds to changes in usage by increasing or decreasing the overall number of nodes available, and is able to scale these independently from the disaggregated storage of timers and state. To visit our IoT factory floor example one last time, these autoscaling capabilities would mean that adding more sensors or increasing their signal frequency wouldn’t require the long operations and provisioning cycles you would have needed in the past.Next, be sure to check out the third and final blog in this series, which aims to compare and contrast Dataflow with some of the other technologies available in the market.
Quelle: Google Cloud Platform

da Agency

Databases that transform businesses—What happened at Google Cloud Next ‘20: OnAir

Week 6 of Google Cloud Next ‘20: OnAir was all about Google Cloud databases and how to choose and use them, no matter where you are in your cloud journey. There was plenty to explore, from deep-dive sessions and demos to feature launches and customer stories. Across it all, what stood out is the strong momentum and adoption across Google Cloud databases for developers and enterprises alike.Google Cloud’s range of databases are designed to help you tackle the unpredictable. Your databases shouldn’t get in the way of innovation and growth, but many legacy, on-prem databases are holding businesses back. We build our databases to meet you at any stage, whether it’s an as-is migration or a brand-new app developed in the cloud.  Key data management announcements this weekThis week, we launched new features aimed at solving the hardest data problems to help our customers run the most mission-critical applications. We kicked off the week with a keynote from Director of Product Management Penny Avril, who talked with social media platform ShareChat about how they met a 500% increase in demand using Cloud Spanner without changing a line of code. We also announced updates to our databases. For Spanner, the Spanner Emulator lets app developers do correctness testing when developing an app. A new C++ client library and increased SQL feature set also add more flexibility. In addition, cloud-native Spanner now offers new multi-region configurations for Asia and Europe with 99.999% availability. NoSQL database service Cloud Bigtable now offers more capabilities, like managed backups for high business continuity and added data protection. And expanded support and SLA for single-node production instances makes it even easier to use Bigtable for all use cases, both large and small. Mobile and web developers use Cloud Firestore to build apps easily, and it now offers a richer query language, C++ client library, and Firestore Unity SDK to make it easy for game developers to adopt Firestore. We are also introducing tools to give you better visibility into usage patterns and performance with Firestore Key Visualizer, which will be coming soon.Cloud SQL, the fully managed service for MySQL, PostgreSQL, and SQL Server, now offers more maintenance controls, cross-region replication, and committed use discounts, providing reliability and flexibility as you migrate to the cloud. For those users running specialized workloads like Oracle, Google Cloud’s Bare Metal Solution enables you to move these workloads within milliseconds of latency to Google Cloud. Our Bare Metal Solution is now available in even more regions and provides a fast track to cloud while lowering overall costs. How customers are building and growing with cloud databasesWe also heard from customers across industries on how they use Google Cloud databases to transform their business, especially in the face of the unpredictable. From The New York Times building a real-time collaborative editor to help publish faster and Khan Academy on how they met the rising demand for online learning to gaming publishers like Colopl supporting massive scale and variable usage through Spanner and ShareChat migrating from Amazon DynamoDB to Spanner for better scale and efficiency at 30% lower costs, it’s exciting to see what they’ve been able to accomplish. Check out data management demosFor data management week, we debuted new interactive demos that let you explore database decisions for yourself. If you’re trying to understand where to start, check out this demo that can help you choose which database is right for you. To see how Cloud SQL lets you achieve high availability, explore this demo. Or learn how you can get a consistent, real-time view of your inventory at scale across channels and regions using Spanner. And take a close look at how Bare Metal Solutions can help you run specialized workloads in the cloud.Go deep with databasesAcross our entire database portfolio, there are sessions to help you better understand each service and what’s new. For SQL Server, MySQL, or Postgres users, check out Getting to Know Cloud SQL for SQL Server or High Availability and Disaster Recovery with Cloud SQL. If it’s cloud-native you’re interested in, sessions like Modernizing HBase Workloads with Cloud Bigtable, Future-proof Your Business for Global Scale and Consistency with Cloud Spanner, or Simplify Complex Application Development Using Cloud Firestore provide deep dives to help you get started.Looking ahead: Application modernizationStay tuned to Next OnAir—next week is all about application modernization. Check out Tuesday’s keynote to learn more about Anthos and how it can help make the most of your on-premises investments and cloud offerings. Of course, we’ll also bring you live technical talks and learning opportunities, aligned with each week’s content. Click “Learn” on the Explore page to find each week’s schedule. Haven’t yet registered for Google Cloud ’20 Next: OnAir? Get started at g.co/cloudnext.
Quelle: Google Cloud Platform

da Agency

Using Cloud Logging as your single pane of glass

Logs are an essential tool for helping to secure your cloud deployments. In the first post in this series, we explored Cloud Identity logs and how you can configure alerts for potentially malicious activity in the Cloud Identity Admin Console to make your cloud deployment more secure. Today, we’ll take it a step further and look at how you can centralize collection of these logs to view activity across your deployment in a single pane of glass. Our best practices for enterprises using Google Cloud Platform (GCP) encourage customers to centralize log management, operations, searching, and analysis in GCP’s Cloud Logging. However, sometimes customers use services and applications that may not automatically or fully log to Cloud Logging. One example of this is Cloud Identity.Fortunately, there’s a way to get Cloud Identity logs into this central repository by using a Cloud Function that executes the open-source GSuite log exporter tool. A Cloud Scheduler job will trigger the execution of this Cloud Function automatically, on a user-defined cadence. Here’s a visual representation of this flow:Google Cloud Professional Services also provides resources that can help you automate the deployment of the GCP tools involved in this solution. Even better, the services used are fully-managed: no work is required post-deployment.Is this solution right for me? Before proceeding, let’s decide if the tools in this post are right for your organization. Cloud Identity Premium has a feature that lets you export Cloud Identity logs straight to BigQuery. This may be sufficient if your organization only needs to analyze the logs in BigQuery. However, you may want to export the logs to Cloud Logging for retention or further processing as part of your normal logging processes.GCP also has a G Suite audit logging feature which automatically publishes some Cloud Identity logs into Cloud Logging. You can explore which Cloud Identity logs this feature covers in the documentation. The G Suite log exporter tool we will explore in this post provides additional coverage for getting Mobile, OAuth Token, and Drive logs into Cloud Logging, and also allows the user to specify exactly which logs they want to ingest from Cloud Identity.If either of these situations are relevant to your organization, keep reading!The tools we useThe G Suite log exporter is an open-source tool developed and maintained by Google Cloud Professional Services. It handles exporting data from Cloud Identity by calling G Suite’s Reports API. It specifies Cloud Logging on GCP as the destination for your logs, grabs the Cloud Identity logs, does some cleanup and reformatting, and writes to Cloud Logging using the Cloud Logging API.One way to run this tool is to spin up a virtual machine using Google Compute Engine. You could import and execute the tool as a Python package and set up a cronjob that runs the tool on a cadence. We even provide a Terraform module that will automate this setup for you. It seems simple enough, but there are some things you must consider if you take this path, including how to secure your VM and what project and VPC it belongs to. An alternative approach is to use Google-managed services to execute this code. Cloud Functions gives you a serverless platform for event-based code execution—no need to spin up or manage any resources to run the code. Cloud Scheduler is Google’s fully managed enterprise-grade cronjob scheduler. You can integrate a Cloud Function with a Cloud Scheduler job so that your code executes automatically on a schedule, per the following steps:Create a Cloud Function that subscribes to a Cloud Pub/Sub topicCreate a Pub/Sub topic to trigger that functionCreate a Cloud Scheduler job that invokes the Pub/Sub triggerRun the Cloud Scheduler job.We also provide open-source examples that will help you take this approach, using a script or a Terraform module. Post-deployment, the Cloud Function will be triggered by the recurring Cloud Scheduler job, and the GSuite log exporter tool will execute indefinitely. That’s it! You now have up-to-date Cloud Identity logs in Cloud Logging. And since we’re using fully-managed GCP services, there’s no further effort required.Customizing the solutionThe open-source examples above can also be customized to fit your needs. Let’s take a look at the one that uses a script.In this example, the default deploy.sh script creates a Cloud Scheduler job that triggers the exporter tool every 15 minutes. But, let’s say your organization needs to pull logs every 5 minutes to meet security requirements. You can simply change the “–schedule” flag in this file so that the exporter tool is fired as often as you’d like. The cadence is defined in unix-cron format.You may also want to customize main.py to control which specific Cloud Identity logs you grab. Our example pulls every log type currently supported by the exporter tool: Admin activity, Google Drive activity, Login activity, Mobile activity, and OAuth Token activity. The log types are defined in the sync_all function call in this file. Simply edit the “applications=” line (Line 34) to customize the log types you export (see below).Next stepsA few minutes after running the script or executing the Terraform module, you will have a Cloud Function deployed that automatically pulls the logs you want from Cloud Identity and puts them into Cloud Logging on a schedule you define. Now you can integrate them into your existing logging processes: send them to Cloud Storage for retention, to BigQuery for analysis, or to a Pub/Sub topic to be exported to a destination such as Splunk.A Cloud Function integrated with a Cloud Scheduler job is a simple but effective way to collect Cloud Identity logs into Cloud Logging, so that your Google Cloud logs live behind a single pane of glass. The fully managed and easy-to-deploy examples we discussed today free up resources and time so your organization can further focus on keeping your cloud safe.
Quelle: Google Cloud Platform