Using Firestore and Apache Beam for data processing

Large scale data processing workloads can be challenging to operationalize and orchestrate. Google Cloud announced the release of a Firestore in Native Mode connector for Apache Beam that makes data processing easier than ever for Firestore users. Apache Beam is a popular open source project that supports large scale data processing with a unified batch and streaming processing model.  It’s portable, works with many different backend runners, and allows for flexible deployment. The Firestore Beam I/O Connector joins BigQuery, Bigtable, and Datastore as Google databases with Apache Beam connectors and is automatically included with theGoogle Cloud Platform IO module of the Apache Beam Java SDK.  The Firestore connector can be used with a variety of Apache Beam backends, including Google Cloud Dataflow. Dataflow, an Apache Beam backend runner, provides a structure for developers to solve “embarrassingly parallel” problems. Mutating every record of your database is an example of such a problem. Using Beam pipelines removes much of the work of orchestrating the parallelization and allows developers to instead focus on the transforms on the data.A practical application of a Firestore Connector for BeamTo better understand the use case for a Beam + Firestore Pipeline, let’s look at an example that illustrates the value of using Google Cloud Dataflow to do bulk operations on a Firestore database. Imagine you have a Firestore database and have a collection group you want to do a high number of operations on; for instance, deleting all documents within a collection group. Doing this on one worker could take a while. What if instead we could use the power of Beam to do it in parallel?This pipeline starts by creating a request for a partition query on a given collectionGroupId. We specify withNameOnlyQuery as it will save on network bandwidth; we only need the name to delete a document. From there, we use a few custom functions. We read the query response to a document object, get the document’s name, and delete a document by that name.Beam utilizes a watermark to ensure exactly-once processing.  As a result, the Shuffle operation stops backtracking over work that is complete already, providing both speed and correctness.While the code to create a partition query is a bit long, it consists of constructing the protobuf request to be sent to Firestore using the generated protobuf builder.Creating  a Partition Query:There are many possible applications for this connector for Google Cloud users. Joining disparate data in a Firestore in Native Mode database, relating data across multiple databases, deleting a large number of entities, writing Firestore data to BigQuery, and more. We’re excited to have contributed this connector to the Apache Beam ecosystem and can’t wait to see how you use the Firestore connector to build the next great thing.Related ArticleAnnouncing a Firestore Connector for Apache Beam and Cloud DataflowGoogle Cloud announces a Firestore connector for Apache Beam, making data processing easier than ever for Firestore users.Read Article
Quelle: Google Cloud Platform

Docker Desktop 4.2 Release: Save Your Battery with Pause / Resume, and Say Goodbye to the Update Pop-up

With Docker Desktop 4.2 we’re excited to introduce Pause / Resume as well as a host of changes to make it easier for you to manage updates. These features are available to Docker Desktop users on any subscription tier.

Save your battery with Pause / Resume

Pause / Resume gives developers the power to pause their current Docker Desktop session and resume work whenever they want, saving resources on their machine while Docker is paused. When you pause Docker Desktop, the current state of your containers is saved in memory and all processes are frozen. This lowers CPU usage and will help with saving your laptop’s battery life. To resume Docker Desktop, click either the Resume button in the menu or type any Docker CLI command in your terminal.

To try out this feature on Docker Desktop 4.2, navigate to the whale menu and click the ‘Pause’ button. The corresponding content sections for Docker Desktop’s left sidebar items (e.g. Containers) will then be covered to clearly denote the Pause state. 

Please note, Pause / Resume is currently not available in Windows container mode.

Say goodbye to the update pop-up

We’ve heard your feedback that the update modal interrupts your workflows and makes it challenging to use Docker Desktop when you need it most. That’s why we’ve done away with the update pop up and introduced a new update settings section in the Docker Dashboard, where you can check for updates and manage your update preferences. We appreciate when our users stay up to date so they get all the latest bug fixes and new features, but we want to make sure that we enable users to do that at a time that’s convenient. 

To summarize what the experience will be like on Docker Desktop 4.2:

A similar badge will appear in the Docker Dashboard settings icon to make it a seamless experience without interrupting your workflow with a modal. Here you can also manage your software update settings.

The `Automatically check for updates` setting is now available for all Docker subscription tiers 

Thanks to all of your positive support of the Docker subscription updates, we’ve been able to focus on delivering more value to all users. In Docker Desktop 4.2 we’ve enabled all users, regardless of subscription tier, to turn off automatically checking for updates. When you disable this setting, all notifications in the whale menu and the app will be disabled and you will have to manually check for updates. Just update to Docker Desktop 4.2 to start using this feature!

We also know that people have different preferences when it comes to downloading updates. For some, the background can take a lot of bandwidth and don’t want it to start when they are busy at work or on Zoom calls, but others would rather have less intervention when it comes to updating. Which is why we’ve put the choice in your hands to decide whether updates should automatically download or not.

We’re considering introducing more settings in the future and would love to know what you think, let us know on our public roadmap item! 

Coming soon

All of the changes described above are available in 4.2 to all Docker Desktop users, including those on Docker Personal. 

We’re also working on two of your highest voted items from our public roadmap: improving Mac filesystem performance, and implementing Docker Desktop for Linux, so watch this space for more news on those in the coming months. And we would love to know what other improvements you would like to see, so please add a thumbs-up emoji to your highest priorities on the roadmap, or create a new roadmap ticket if your idea isn’t already there.
The post Docker Desktop 4.2 Release: Save Your Battery with Pause / Resume, and Say Goodbye to the Update Pop-up appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Save the Date: Next Community All Hands on December 9th

We’re one month away from our next Community All Hands event, on December 9th at 8am PST/5pm CET. This is a unique opportunity for Docker staff, Captains, and the broader Docker community to come together for live company updates, product updates, demos, community shout-outs and Q&A. The last all-hands gathered more than 2,000 attendees from around the world and we hope to double that this time around. 

The theme for this edition is Innovation and Experimentation, so bring your most innovative projects to share with the community. We will be presenting our latest developments and some exciting demos to experiment with. 

Here are five reasons you should attend the next Community All Hands:

Get the inside scoop on Docker’s latest product developmentsYou’ll get the opportunity to hear directly from the product team about the latest developments, including product strategy, new features, roadmap.Meet the Docker teamWe’ll be there to answer your questions, hear your feedback, and engage in lively discussion about the latest developments in the Docker community.Work for DockerThe Docker team is growing fast and we’re looking for people who are passionate about building great software. You’ll have the opportunity to learn more about the Docker team and the open positions we’re hiring for.Get inspired by the community We’ll be showcasing some of the most innovative projects that are being built with Docker. You’ll have the opportunity to learn about what others are doing with Docker and get inspired for your own projects.Get your hands on some cool swagWe’ll have some awesome Docker gear for you to take home. Stay tuned for more details!

Our Call for Speakers will be open until November 19th. If you would like to submit a proposal to speak, please visit our Call for Speakers page.
The post Save the Date: Next Community All Hands on December 9th appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Vereinfachte CI/CD-Konfiguration für AWS Serverless-Anwendungen und Ihr bevorzugtes CI/CD-System – Allgemeine Verfügbarkeit

Mit einer neuen Pipeline-Konfigurationsfunktion für Serverless-Anwendungen können Sie jetzt sichere Pipelines für die kontinuierliche Integration und Bereitstellung (CI/CD) erstellen, die den bewährte Methoden Ihrer Organisation entsprechen. AWS Serverless Application Model Pipelines (AWS SAM Pipelines) ist eine neue Funktion von AWS SAM CLI, mit der Sie innerhalb von Minuten auf die Vorteile von CI/CD zugreifen können, z. B. Beschleunigung der Bereitstellungshäufigkeit, Verkürzung der Vorlaufzeit für Änderungen und Reduzierung von Bereitstellungsfehlern. AWS SAM Pipelines enthält eine Reihe von Standard-Pipeline-Vorlagen für gängige CI/CD-Systeme wie CloudBees CI/Jenkins, GitLab CI/CD, GitHub Actions, Bitbucket Pipelines und AWS CodeBuild/CodePipeline, die den bewährten Praktiken zur Bereitstellung von AWS entsprechen. AWS SAM CLI ist ein Entwicklertool, das das Erstellen, lokale Testen, Verpacken und Bereitstellen von Serverless-Anwendungen erleichtert.
Quelle: aws.amazon.com