Cloud Computing Köln - Seite 3199 von 7121 - Neues zu Cloud Computing, Internet of Things und Technologien

At Docker, we are always looking for ways to make developers’ lives easier either directly or by working with our partners. Improving developer productivity is a core benefit of using Docker products and recently one of our partners made an announcement that makes developing cloud-native apps easier.

AWS announced that its customers can now configure their Amazon Elastic Container Service (ECS) applications deployed in Amazon Elastic Compute Cloud (EC2) mode to access Amazon Elastic File Storage (EFS) file systems. This is good news for Docker developers who use Amazon ECS. It means that Amazon ECS now natively integrates with Amazon EFS to automatically mount shared file systems into Docker containers. This allows you to deploy workloads that require access to shared storage such as machine learning workloads, containerizing legacy apps, or internal DevOps workloads such as GitLab, Jenkins, or Elasticsearch.

The beauty of containerizing your applications is to provide a better way to create, package, and deploy software across different computing environments in a predictable and easy-to-manage way. Containers were originally designed to be stateless and ephemeral (temporary). A stateless application is one that neither reads nor stores information about its state from one time that it is run to the next. A stateful application, on the other hand, can remember some things about its state each time it runs.

Maintaining state in an app means finding a way to connect containers to stateful storage. For example, if you open up your weather app on your mobile device, it remembers your home city as the weather app maintains state. The only way to containerize applications that require state is to connect containers to stateful, persistent storage.

“Docker and AWS are collaborating on making the right workloads more easily deployed as stateful containerized applications. Docker’s industry-leading container technology including Docker Desktop and Docker Hub are integral to advancing developer workflows for modern apps. Our customers can now deploy and run Docker containers seamlessly on Amazon ECS and Amazon EFS, enabling development teams to ship apps faster,” according to Justin Graham, Vice President of Products for Docker.

If you are a developer who would like to deploy workloads that require access to shared external storage, highly-available regional storage, or high-throughput storage then the combination of Amazon ECS and Amazon EFS is your answer. Developers familiar with Amazon ECS can now use the ECS task definition to specify the file system ID and specific directory that they would like to mount on one or more containers in their task. ECS takes care of mounting the file-system on the container so that you can focus on your applications without having to worry about configuring infrastructure.

If you are interested in how to actually deploy a stateful container-based application, AWS’ Martin Beeby has a great blog post that walks through how to configure Amazon EFS to add state to your containers running on Amazon ECS. Developers who are interested in learning more about how to get started with Docker can expand their understanding with these additional resources: Docker Desktop and Docker Hub.

The post Deploy Stateful Docker Containers with Amazon ECS and Amazon EFS appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

11. April 2020

da Agency

How do I move data from MySQL to BigQuery?

In a market where streaming analytics is growing in popularity, it’s critical to optimize data processing so you can reduce costs and ensure data quality and integrity. One approach is to focus on working only with data that has changed instead of all available data. This is where change data capture (CDC) comes in handy. CDC is a technique that enables this optimized approach. Those of us working on Dataflow, Google Cloud’s streaming data processing service, developed a sample solution that lets you ingest a stream of changed data coming from any kind of MySQL database on versions 5.6 and above (self-managed, on-prem, etc.), and sync it to a dataset in BigQuery. We made this solution available within the public repository of Dataflow templates. You can find instructions on using the template in the README section of the GitHub repo. CDC provides a representation of data that has changed in a stream, allowing computations and processing to focus specifically on changed records. CDC can be applied for many use cases. Some examples include replication of a critical database, optimization of a real-time analytics job, cache invalidation, synchronization between a transactional data store and a data warehouse-type store, and more. How Dataflow’s CDC solution moves data from MySQL to BigQueryThe deployed solution, shown below, works with any MySQL database, which is monitored by a connector we developed based on Debezium. The connector stores table metadata using Data Catalog (Google Cloud’s scalable metadata management service) and pushes updates to Pub/Sub (Google Cloud-native stream ingestion and messaging technology). A Dataflow pipeline then takes those updates from Pub/Sub and syncs the MySQL database with a BigQuery dataset.This solution relies on Debezium, which is an excellent open source tool for CDC. We developed a configurable connector based on this technology that you can run locally or on your own Kubernetes environment to push change data to Pub/Sub.Click to enlargeUsing the Dataflow CDC solutionDeploying the solution consists of four steps:Deploy your database (nothing to do here if you already have a database)Create Pub/Sub topics for each of the tables you want to exportDeploy our Debezium-based connectorStart the Dataflow pipeline to consume data from Pub/Sub and synchronize to BigQueryLet’s suppose you have a MySQL database running in any environment. For each table in the database that you want to export, you must create a Pub/Sub topic and a corresponding subscription for that topic.Once the Pub/Sub topics and the database are in place, run the Debezium connector. The connector can run in many environments: locally built from source, via a Docker container, or on a Kubernetes cluster. For instructions on running the Debezium connector and the solution in general, check out the README for detailed instructions.Once the Debezium connector starts running and capturing changes from MySQL, it will push them to Pub/Sub. Using Data Catalog, it will also update schemas for the Pub/Sub topic that corresponds to each MySQL table.With all of these pieces in place, you can launch the Dataflow pipeline to consume the change data from Pub/Sub and synchronize it to BigQuery tables. The Dataflow job can be launched from the command line. Here’s what it looks like once you launch it:Once the connector and pipeline are running, you just need to monitor their progress, and make sure that it’s all going smoothly.Get started todayGot a use case that aligns with Dataflow’s CDC capabilities? For example, optimization of an existing real-time analytics job. If so, try it out! First, use this code to get started with building your first CDC pipeline in Dataflow today. And share your feedback with the Dataflow team in the GitHub issue tracker.At Google Cloud, we’re excited to bring CDC as an incredibly valuable technique to optimize streaming data analytics. We look forward to seeing both development and feedback with these new capabilities for Dataflow.
Quelle: Google Cloud Platform