da Agency

Leading from open: How military veterans view open culture at Red Hat

Red Hat’s culture is built on the open source principles of transparency, collaboration and inclusion, where the best ideas can come from anywhere and anyone. You might not immediately think that this culture has a lot of overlap with that of the military, which may appear to be hierarchical and rely on command and control, but Red Hatters from our Military Veterans Diversity and Inclusion Community would disagree. 
Quelle: CloudForms

da Agency

Scale your data science workflows with the Vertex AI Workbench notebook executor

When solving a new ML problem, it’s common to start by experimenting with a subset of your data in a notebook environment. But if you want to execute a long-running job, add accelerators, or run multiple training trials with different input parameters, you’ll likely find yourself copying code over to a Python file to do the actual computation. That’s why we’re excited to announce the launch of the notebook executor, a new feature of Vertex AI Workbench that allows you to schedule notebooks ad hoc, or on a recurring basis. With the executor, your notebook is run cell by cell on Vertex AI Training. You can seamlessly scale your notebook workflows by configuring different hardware options, passing in parameters you’d like to experiment with, and setting an execution schedule, all via the Console UI or the notebooks API. Built to ScaleImagine you’re tasked with building a new image classifier. You start by loading a portion of the dataset into your notebook environment and running some analysis and experiments on a small machine. After a few trials, your model looks promising, so you want to train on the full image dataset. With the notebook executor, you can easily scale up model training by configuring a cluster with machine types and accelerators, such as NVIDIA GPUs, that are much more powerful than the current instance where your notebook is running.Your model training gets a huge performance boost from adding a GPU, and you now want to run a few extra experiments with different model architectures from TensorFlow Hub. For example, you can train a new model using feature vectors from various architectures, such as Inception, ResNet, or MobileNet, all pretrained on the ImageNet dataset. Using these feature vectors with the Keras Sequential API is simple; all you need to do is pass the TF Hub URL for the particular model to hub.KerasLayer.Instead of running these trials one by one in the notebook, or making multiples copies of your notebook (inceptionv3.ipynb, resnet50.ipynb, etc) for each of the different TF Hub URLs, you can experiment with different architectures by using a parameter tag. To use this feature, first select the cell you want to parameterize. Then click on the gear icon in the top right corner of your notebook.Type “parameters” in the Add Tag box and hit Enter. Later when configuring your execution, you’ll pass in the different values you want to test.In this example, we create a parameter called feature_extractor_model, and we’ll pass in the name of the TF hub model we want to use when launching the execution. That model name will be substituted into the tf_hub_uri variable, which is then passed to the hub.KerasLayer, as shown in the screenshot above.  After you’ve discovered the optimal model architecture for your use case, you’ll want to track the performance of your model in production. You can create a notebook that pulls the most recent batch of serving data that you have labels for, gets predictions, and computes the relevant metrics. By scheduling these jobs to execute on a recurring basis, you’ve created a lightweight monitoring system that tracks the quality of your model predictions over time. The executor supports your end-to-end ML workflow, making it easy to scale up or scale out notebook experiments written with Vertex AI Workbench.Configuring ExecutionsExecutions can be configured through the Cloud Console UI or the Notebooks API.In your notebook, click on the Executor icon.In the side panel on the right specify the configuration for your job, such as the machine type and the environment. You can select an existing image, or provide your own custom docker container image.If you’ve added parameter tags to any of your notebook cells, you can pass in your parameter values to the executor.Finally, you can choose to run your notebook as a one time execution, or schedule recurring executions.Then click SUBMIT to launch your job.In the EXECUTIONS tab, you’ll be able to track the status of your notebook execution.When your execution completes, you’ll be able to see the output of your notebook by clicking VIEW RESULT.You can see that an additional cell was added with the comment # Parameters, that overrides the default value for feature_extractor_model, with the value we passed in at execution time. As a result, the feature vectors used for this execution came from a ResNet50 model instead of an Inception model.What’s Next?You now know the basics of how to use the notebook executor to train with a more performant hardware profile, test out different parameters, and track model performance over time. If you’d like to try out an end-to-end example, check out this tutorial. It’s time to run some experiments of your own!
Quelle: Google Cloud Platform

da Agency

Announcing Vertex Pipelines general availability

Today, we’re incredibly excited to announce the general availability of Vertex Pipelines.One of the best ways to scale your machine learning (ML) workflows is to run them as a pipeline, where each pipeline step is a distinct piece of your ML process. Pipelines are a great tool for productionizing, sharing, and reliably reproducing ML workflows across your organization. They are also the key to MLOps – with Pipelines you can build systems to automatically retrain and deploy models. In this post, we’ll show what you can do with Vertex Pipelines, and we’ll end by sharing a sample pipeline to help you get started.Vertex Pipelines in a nutshellLet’s briefly discuss what an ML pipeline does. A machine learning pipeline is an ML workflow encapsulated as a series of steps, also called components. Each step in a pipeline is a container, and the output of each step can be an input to the next step. This presents two challenges:For this to work, you need a way to convert individual pipeline steps to containersThis will require setting up infrastructure to run your pipeline at scaleTo address the first challenge, there are some great open source libraries that handle converting pipeline steps to containers and managing the flow of input and output artifacts throughout your pipeline, allowing you to focus on building out the functionality of each pipeline step. Vertex Pipelines supports two popular open source libraries – Kubeflow Pipelines (KFP) and TensorFlow Extended (TFX). This means you can define your pipeline using one of these libraries and run it on Vertex Pipelines.Second, Vertex Pipelines is entirely serverless. When you upload and run your KFP or TFX pipelines, Vertex AI will handle provisioning and scaling the infrastructure to run your pipeline. This means you’ll only pay for the resources used while your pipeline runs; and your data scientists get to focus on ML without worrying about infrastructure. Vertex Pipelines integrates with other tools in Vertex AI and Google Cloud: you can import data from BigQuery, train models with Vertex AI, store pipeline artifacts in Cloud Storage, get model evaluation metrics, and deploy models to Vertex AI endpoints, all within your Vertex Pipeline steps. To make this easy, we’ve created a library of pre-built components for Vertex Pipelines. These pre-built components help simplify the process of using other parts of Vertex AI in your pipeline steps, like creating a dataset or training an AutoML model. To use them, import the pre-built component library and use components from the library directly in your pipeline definition. As an example, below is a pipeline that creates a Vertex AI dataset pointing to data in BigQuery, trains an AutoML model, and deploys the trained model to an endpoint if its accuracy is above a certain threshold:ML Metadata & Vertex PipelinesWith output generated from each step of a pipeline execution, it’s important to have a mechanism for tracking the artifacts and metrics created across pipeline runs. This becomes especially useful when you have multiple people on your team involved in developing and running a pipeline, or are managing multiple pipelines for different ML tasks. To help with this, Vertex Pipelines integrates directly with Vertex ML Metadata for automatic artifact, lineage, and metric tracking.You can inspect pipeline metadata in the Vertex AI console, and through the Vertex AI SDK. To see metadata and artifacts in the console, start by clicking on the “Expand artifacts” slider when looking at your pipeline graph. You can then click on individual artifacts to view details and see where each artifact is stored:When looking at output artifacts, it’s also helpful to understand an individual artifact in the larger context of your pipeline. To do this, Vertex Pipelines offers lineage tracking. When looking at an artifact in the console, click on the “View lineage” button. As an example, for the endpoint below, we can see the model that is deployed to the endpoint, and the dataset used to train that model. We can also see the pipeline steps that generated each artifact in this graph:There are several ways to interact with pipeline metadata programmatically. Using the Vertex ML Metadata API, you can query for any artifacts or executions in your metadata store by their properties or lineage. You can also use the get_pipeline_df method from the Vertex AI SDK to create a Pandas DataFrame with metrics from each of your pipeline runs. There are also SDK methods for getting artifact lineage and filtering artifacts, which you can use to create custom dashboards to track your pipelines.Building a sample pipelineTo see Vertex Pipelines in action, let’s look at an example built with the Kubeflow Pipelines SDK. You can find the full pipeline code for this example in this codelab, and here we’ll show a few highlights. Our sample pipeline will make use of Google Cloud’s pre-built components, and will do the following:Create a dataset in Vertex AITrain a custom model on that datasetRun a batch prediction on the trained modelTo build and run this pipeline, we’ll first import a few Python packages:We’re using three libraries to build and run this pipeline on Vertex AI:The Kubeflow Pipelines SDK to build our components and connect them together into a pipelineThe Vertex AI SDK to run our pipeline on Vertex PipelinesThe Google Cloud components library to make use of pre-built components for interacting with various Google Cloud servicesBecause we’re making use of first party pre-built components, we don’t need to write boilerplate code to perform each of these tasks. Instead, we can pass configuration variables to the components directly in our pipeline definition. You can see the full definition in the codelab, and we’ve shown some highlights below:This pipeline first creates a dataset in Vertex AI using the TabularDatasetCreateOp component, passing in the BigQuery source table of the dataset. The dataset created will then be passed to our CustomContainerTrainingJobRunOp component and used in our scikit-learn model training job. We’ve passed configuration parameters that point to a container in Container Registry where we’ve deployed our scikit-learn training code. The output of this component is a model in Vertex AI. In the last component of this pipeline, we run a batch prediction job on this model by providing a CSV file with examples we’d like to get predictions on.When we compile and run this pipeline on Vertex AI, we can inspect the pipeline graph in the console as it runs:Start building with Vertex PipelinesReady to run your own scalable ML pipelines on Vertex AI? Check out the following resources to get started:Vertex Pipelines documentationOfficial sample pipelines on GitHubCheck out this codelab for an introduction to Vertex Pipelines, and this one to understand how Vertex Pipelines works with Vertex ML Metadata.Related ArticleUse Vertex Pipelines to build an AutoML classification end-to-end workflowHow you can use Vertex Pipelines to build an end-to-end ML workflow for training a custom model using AutoMLRead Article
Quelle: Google Cloud Platform

da Agency

Using Firestore and Apache Beam for data processing

Large scale data processing workloads can be challenging to operationalize and orchestrate. Google Cloud announced the release of a Firestore in Native Mode connector for Apache Beam that makes data processing easier than ever for Firestore users. Apache Beam is a popular open source project that supports large scale data processing with a unified batch and streaming processing model.  It’s portable, works with many different backend runners, and allows for flexible deployment. The Firestore Beam I/O Connector joins BigQuery, Bigtable, and Datastore as Google databases with Apache Beam connectors and is automatically included with theGoogle Cloud Platform IO module of the Apache Beam Java SDK.  The Firestore connector can be used with a variety of Apache Beam backends, including Google Cloud Dataflow. Dataflow, an Apache Beam backend runner, provides a structure for developers to solve “embarrassingly parallel” problems. Mutating every record of your database is an example of such a problem. Using Beam pipelines removes much of the work of orchestrating the parallelization and allows developers to instead focus on the transforms on the data.A practical application of a Firestore Connector for BeamTo better understand the use case for a Beam + Firestore Pipeline, let’s look at an example that illustrates the value of using Google Cloud Dataflow to do bulk operations on a Firestore database. Imagine you have a Firestore database and have a collection group you want to do a high number of operations on; for instance, deleting all documents within a collection group. Doing this on one worker could take a while. What if instead we could use the power of Beam to do it in parallel?This pipeline starts by creating a request for a partition query on a given collectionGroupId. We specify withNameOnlyQuery as it will save on network bandwidth; we only need the name to delete a document. From there, we use a few custom functions. We read the query response to a document object, get the document’s name, and delete a document by that name.Beam utilizes a watermark to ensure exactly-once processing.  As a result, the Shuffle operation stops backtracking over work that is complete already, providing both speed and correctness.While the code to create a partition query is a bit long, it consists of constructing the protobuf request to be sent to Firestore using the generated protobuf builder.Creating  a Partition Query:There are many possible applications for this connector for Google Cloud users. Joining disparate data in a Firestore in Native Mode database, relating data across multiple databases, deleting a large number of entities, writing Firestore data to BigQuery, and more. We’re excited to have contributed this connector to the Apache Beam ecosystem and can’t wait to see how you use the Firestore connector to build the next great thing.Related ArticleAnnouncing a Firestore Connector for Apache Beam and Cloud DataflowGoogle Cloud announces a Firestore connector for Apache Beam, making data processing easier than ever for Firestore users.Read Article
Quelle: Google Cloud Platform

da Agency

Docker Desktop 4.2 Release: Save Your Battery with Pause / Resume, and Say Goodbye to the Update Pop-up

With Docker Desktop 4.2 we’re excited to introduce Pause / Resume as well as a host of changes to make it easier for you to manage updates. These features are available to Docker Desktop users on any subscription tier.

Save your battery with Pause / Resume

Pause / Resume gives developers the power to pause their current Docker Desktop session and resume work whenever they want, saving resources on their machine while Docker is paused. When you pause Docker Desktop, the current state of your containers is saved in memory and all processes are frozen. This lowers CPU usage and will help with saving your laptop’s battery life. To resume Docker Desktop, click either the Resume button in the menu or type any Docker CLI command in your terminal.

To try out this feature on Docker Desktop 4.2, navigate to the whale menu and click the ‘Pause’ button. The corresponding content sections for Docker Desktop’s left sidebar items (e.g. Containers) will then be covered to clearly denote the Pause state. 

Please note, Pause / Resume is currently not available in Windows container mode.

Say goodbye to the update pop-up

We’ve heard your feedback that the update modal interrupts your workflows and makes it challenging to use Docker Desktop when you need it most. That’s why we’ve done away with the update pop up and introduced a new update settings section in the Docker Dashboard, where you can check for updates and manage your update preferences. We appreciate when our users stay up to date so they get all the latest bug fixes and new features, but we want to make sure that we enable users to do that at a time that’s convenient. 

To summarize what the experience will be like on Docker Desktop 4.2:

A similar badge will appear in the Docker Dashboard settings icon to make it a seamless experience without interrupting your workflow with a modal. Here you can also manage your software update settings.

The `Automatically check for updates` setting is now available for all Docker subscription tiers 

Thanks to all of your positive support of the Docker subscription updates, we’ve been able to focus on delivering more value to all users. In Docker Desktop 4.2 we’ve enabled all users, regardless of subscription tier, to turn off automatically checking for updates. When you disable this setting, all notifications in the whale menu and the app will be disabled and you will have to manually check for updates. Just update to Docker Desktop 4.2 to start using this feature!

We also know that people have different preferences when it comes to downloading updates. For some, the background can take a lot of bandwidth and don’t want it to start when they are busy at work or on Zoom calls, but others would rather have less intervention when it comes to updating. Which is why we’ve put the choice in your hands to decide whether updates should automatically download or not.

We’re considering introducing more settings in the future and would love to know what you think, let us know on our public roadmap item! 

Coming soon

All of the changes described above are available in 4.2 to all Docker Desktop users, including those on Docker Personal. 

We’re also working on two of your highest voted items from our public roadmap: improving Mac filesystem performance, and implementing Docker Desktop for Linux, so watch this space for more news on those in the coming months. And we would love to know what other improvements you would like to see, so please add a thumbs-up emoji to your highest priorities on the roadmap, or create a new roadmap ticket if your idea isn’t already there.
The post Docker Desktop 4.2 Release: Save Your Battery with Pause / Resume, and Say Goodbye to the Update Pop-up appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

da Agency

Save the Date: Next Community All Hands on December 9th

We’re one month away from our next Community All Hands event, on December 9th at 8am PST/5pm CET. This is a unique opportunity for Docker staff, Captains, and the broader Docker community to come together for live company updates, product updates, demos, community shout-outs and Q&A. The last all-hands gathered more than 2,000 attendees from around the world and we hope to double that this time around. 

The theme for this edition is Innovation and Experimentation, so bring your most innovative projects to share with the community. We will be presenting our latest developments and some exciting demos to experiment with. 

Here are five reasons you should attend the next Community All Hands:

Get the inside scoop on Docker’s latest product developmentsYou’ll get the opportunity to hear directly from the product team about the latest developments, including product strategy, new features, roadmap.Meet the Docker teamWe’ll be there to answer your questions, hear your feedback, and engage in lively discussion about the latest developments in the Docker community.Work for DockerThe Docker team is growing fast and we’re looking for people who are passionate about building great software. You’ll have the opportunity to learn more about the Docker team and the open positions we’re hiring for.Get inspired by the community We’ll be showcasing some of the most innovative projects that are being built with Docker. You’ll have the opportunity to learn about what others are doing with Docker and get inspired for your own projects.Get your hands on some cool swagWe’ll have some awesome Docker gear for you to take home. Stay tuned for more details!

Our Call for Speakers will be open until November 19th. If you would like to submit a proposal to speak, please visit our Call for Speakers page.
The post Save the Date: Next Community All Hands on December 9th appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/