AI in Depth: Serving a PyTorch text classifier on AI Platform Serving using custom online prediction

Earlier this week, we explained in detail how you might build and serve a text classifier in TensorFlow. Today, we’ll provide a new explainer on how to build a similar classifier in PyTorch, another machine learning framework. In today’s blog post, we’ll explain how to implement the same model using PyTorch, and deploy it to AI Platform Serving for online prediction. We will reuse the preprocessing implemented in Keras in the previous blog post. The code for this example can be found in this Notebook.AI Platform ML Engine is a serverless, NoOps product that lets you train and serve machine learning models at scale. These models can then be served as REST APIs for online prediction. The AI Platform Serving automatically scales to adjust to any throughput, and provides secure authentication to its REST endpoints.To help maintain affinity of preprocessing between training and serving, AI Platform Serving now enables users to customize the prediction routine that gets called when sending prediction requests to their model deployed on AI Platform Serving. This feature allows you to upload a Custom Model Prediction class, along with your exported model, to apply custom logic before or after invoking the model for prediction.In other words, we can now leverage AI Platform Serving to execute arbitrary Python code, breaking the typical and previous coupling with TensorFlow. This change enables you to pick the best framework for the job, or even combine multiple frameworks into a single application. For example, we can use Keras APIs for their easy-to-use text pre-processing methods, and combine them with PyTorch for the actual machine learning model. This combination of frameworks is precisely what we’ll discuss in this blog post.For more details on text classification, the Hacker News dataset used in the example, and the text preprocessing logic, refer to the Serving a Text Classifier with Preprocessing using AIPlatform Serving blog post.Building a PyTorch text classification modelYou can begin by implementing your TorchTextClassifier model class in the torch_model.py module. As shown in the following code block, we implement the same text classification model architecture described in this post, which consists of an Embedding layer, Dropout layer, followed by two Conv1d and Pooling Layers, then a Dense layer with Softmax activation at the end.Loading and preprocessing dataThe following code prepares both the training and evaluation data. Note that, you use both fit() and transform() with the training data, while you only use transform() with the evaluation data, to make use of the tokenizer generated from the training data. The created train_texts_vectorized and eval_texts_vectorized objects will be used to train and evaluate our text classification model respectively.The implementation of TextPreprocessor class, which uses Keras APIs, is described in Serving a Text Classifier with Preprocessing using AI Platform Serving blog post.Now you need to save the processor object—which includes the tokenizer generated from the training data—to be used when serving the model for prediction. The following code dumps the object to a new processor_state.pkl file.Training and saving the PyTorch modelThe following code snippet shows you how to train your PyTorch model. First, you create an object of the TorchTextClassifier, according to your parameters. Second, you implement a training loop, in which each iteration you predictions from your model (y_pred) given the current training batch, compute the loss using cross_entropy, and backpropagation using loss.backward() and optimizer.step(). After NUM_EPOCH epochs, the trained model is saved to torch_saved_model.pt file.Implementing the Custom Prediction classIn order to apply a custom prediction routine, which includes both preprocessing and postprocessing, you need to wrap this logic in a Custom Model Prediction class. This class, along with the trained model and its corresponding preprocessing object, will be used to deploy the AI Platform Serving microservices. The following code shows how the Custom Model Prediction class (CustomModelPrediction) for our text classification example is implemented in the model_prediction.py module.Deploying to AI Platform servingUploading the artifacts to Cloud StorageNext, you’ll want to upload your artifacts to Cloud Storage, as follows:Your saved (trained) model file: trained_saved_model.pt (see Training and Saving the PyTorch model).Your pickled preprocessing objects (which contain the state needed for data transformation prior to prediction): processor_state.pkl. As described in the previous, Keras-based post, the processor_state.pkl object includes the tokenizer generated from the training data.Second, you need to upload a Python package including all the classes you’ll need for prediction (preprocessing, model classes, and post-processing, if any). In this example, you need to create a `pip`-installable tar file that includes torch_model.py, model_prediction.py, and preprocess.py. To begin, create the following setup.py file:The setup.py file includes a list of the PyPI packages you need to `pip install` and use for prediction in the REQUIRED_PACKAGES variable.Because we are deploying a model implemented by PyTorch, we need to include ‘torch’ in REQUIRED_PACKAGES. Now, you can create the package by running the following command:This will create a `.tar.gz` package under /dist directory. The name of the package will be `$name-$version.tar.gz` where `$name` and `$version` are the ones specified in setup.py.Once you have successfully created the package, you can upload it to Cloud Storage:Deploying the model to AI Platform ServingLet’s define the model name, the model version, and the AI Platform Serving runtime (which corresponds to a TensorFlow version) required for deploying the model.First, you create a model in AI Platform Serving by running the following gcloud command:Second, you create a model version using the following gcloud command, in which you specify the location of the model and preprocessing object (–origin), the location the package(s) including the scripts needed for your prediction (–package-uris), and a pointer to you Custom Model prediction class (–prediction-class).This should take 1-2 minutes.After deploying the model to AI Platform Serving, you can invoke the model for prediction using the code described in previous Keras-based blog post .Note that the client of our REST API does not need to know whether the service was implemented in TensorFlow or in PyTorch. In either case, the client should send the same request, and receive a response of the same form.ConclusionAlthough AI Platform initially provided only support for TensorFlow, it is now evolving into a platform that supports multiple frameworks. You can now deploy models using TensorFlow, PyTorch, or any Python-based ML framework, since AI Platform Serving supports custom prediction Python code, available in beta. This post demonstrates that you can flexibly deploy a PyTorch text classifier, which utilizes text preprocessing logic implemented in using Keras.Feel free to reach out @GCPcloud if there are still features or other frameworks you’d like to train or deploy on AI Platform Serving.Next stepsTo learn more about AI Platform serving custom online prediction, read this blog post.To learn more about machine learning on GCP, take this course.To try out the code, run this Notebook.
Quelle: Google Cloud Platform

American Cancer Society uses Google Cloud machine learning to power cancer research

Among the most promising and important applications of machine learning is finding better ways to diagnose and treat life threatening conditions, including diseases such as cancer that cut far too many lives short. In the United States, cancer is the second most common cause of death and accounts for nearly one in four deaths. Prevention and early detection are critical to improving survival, but there remains much that medical professionals do not understand about lifestyle factors, diagnosis, and treatment of specific subtypes of cancer.The American Cancer Society is using Google Cloud to reinvent the way data are analyzed so they can save more lives. For the past few decades ACS has conducted the Cancer Prevention Study-II (CPS-II) Nutrition cohort, a prospective study of more than 184,000 American men and women, to explore how factors such as height, weight, demographic characteristics, personal and family history can affect cancer etiology and prognosis.Mia M. Gaudet, PhD, Scientific Director of Epidemiology Research at ACS, was able to use an end-to-end ML pipeline built on Google Cloud to perform deep analysis of breast cancer tissue samples, the most commonly diagnosed type of cancer among women and the second leading cause of cancer death.After obtaining medical records and surgical tissue samples for 1,700 CPS-II study participants who were diagnosed with breast cancer from hundreds of hospitals throughout the U.S., Dr. Gaudet studied high-resolution images of the tumor tissue in an effort to determine what lifestyle, medical, and genetic factors are related to molecular subtypes of breast cancer, and whether different features in the breast cancer tissue translate to a better prognosis.She faced a few technical challenges in analyzing the 1,700 images of breast tumor tissue:They were captured in a high-resolution, uncompressed and proprietary format—up to 10GB each. Image conversion would be exceedingly costly and time consuming.Even if the images were converted to a usable format, it would take a team of highly trained pathologists up to three years to analyze all 1,700, and at significant cost.Analysis would be subject to human fatigue and bias, and some patterns might not be detectable by humans at all.How Slalom used Cloud ML Engine to help Dr. Gaudet complete her researchTo overcome these challenges, Dr. Gaudet and ACS teamed up with Slalom, a Cloud premier partner, to facilitate deep learning at scale. Quality of preprocessing standardization was critical and the images needed to be translated consistently, with colors normalized.The interpretation of colors across images was standardized through the reduction of color variance and every image was broken into evenly sized tiles to distribute the workload and optimize the data structure required to train the models.Slalom used GCP to build an end-to-end machine learning pipeline, including preprocessing, feature engineering, and clustering:Cloud Machine Learning Engine (Cloud ML Engine) preprocessing enabled model training and batch prediction.Images were stored using Cloud Storage.Compute Engine orchestrated image conversion and initiated Cloud ML Engine training and prediction jobs in the correct sequence.Using Keras with a TensorFlow backend for prototyping, Slalom created an auto-encoder model. It then used distributed training on Cloud ML Engine to convert the images into feature vectors that represent patterns in the images as a sequence of numbers.The features were then clustered with TensorFlow, once again using Cloud ML Engine. The result was a set of cluster assignments, one for each tile in the image, that American Cancer Society plans on using in follow-up analyses.With this approach, analysis was completed in only three months—twelve times faster than projected—and with a higher degree of accuracy and consistency. The analysis found interesting results: it isolated potentially significant patterns in the cancer tissue that might help inform risk factors and prognosis in the future.”By leveraging Cloud ML Engine to analyze cancer images, we’re gaining more understanding of the complexity of breast tumor tissue and how known risk factors lead to certain patterns,” said Gaudet.ACS now has established processes and a cloud infrastructure that will be reusable on similar projects to come. We’re enormously proud that our technology is helping medical professionals who are working tirelessly to prevent cancer deaths and improve outcomes.For more information on Cloud ML Engine, visit our website.
Quelle: Google Cloud Platform

Containing our enthusiasm: All the Kubernetes security news from Google Cloud Next ‘19

At Google, we like to think of container security in three pillars: Secure to develop (infrastructure security protecting identities, secrets and networks); secure to build and deploy (vulnerability-free images, verification of what you deploy); and secure to run (isolating workloads, scaling, and identifying malicious containers in production). These pillars cover the entire lifecycle of a container, and help ensure end-to-end security.We’ve been hard at work to make it easier for you to ensure security as you develop, build, deploy, and run containers, with new products and features in Google Kubernetes Engine and across Google Cloud. Here’s what we recently announced at Next ‘19, and how you can use these for your container deployments—so there’s less cryptojacking, and more time for whale watching, as it were.Secure to develop, by making it easier to manage identities and secretsA frequent pain point in GKE is authentication from a container workload to another service on Google Cloud, such as Cloud SQL. Traditionally, there have been two main ways to do this. First, by over-provisioning permissions—for example, using the node’s built-in service account to authenticate, but this creates unnecessary security risks in the case that the pod is compromised. Second, by creating a new service account identity, and storing its key as a secret and injecting that secret into a pod—a cumbersome solution.With the upcoming workload identity feature, you can use Google IAM service accounts from Kubernetes pods without having to manage any secrets. Using workload identity, Kubernetes service accounts are associated with Google service accounts and, when a pod with that Kubernetes service account uses the application’s default credentials, a token exchange occurs and the pod is given a short-lived token for the specified service account. This helps you better scope which workloads can access which other services within your infrastructure.Another pain point has been granting and revoking users’ access through Kubernetes RBAC. Kubernetes RBAC is a core component of Kubernetes and important for fine-grained access control. However, you were previously only able to grant roles to GCP user accounts or Cloud IAM service accounts. Now, you can use Google Groups for GKE in beta. Any Google Group can now be used in an RBAC rule in GKE, provided a Cloud Identity administrator has enabled the group for use in access control rules. This allows you to use existing groups to provide access to large sets of users with a single rule, while ensuring that sensitive groups used exclusively for email distribution remain private. Google Groups for GKE greatly simplifies RBAC and account management.As for your secrets, we released another security measure to protect these a few months ago, with application-layer secrets encryption (beta), which lets you use a key in Cloud KMS to protect secrets at rest in GKE.Secure to build and deploy, with a well-protected software supply chainAnother area of focus at Next this year was to round out our secure software supply chain offerings, including the forthcoming general availability of Container Registry vulnerability scanning and Binary Authorization.Container Registry vulnerability scanning looks at the images in your private registry for known common vulnerabilities and exposures (CVEs) from multipleCVE databases. It displays the results in your registry, including whether a fix is available, so that you can take action. It performs this scan when a new image is added to the registry, as well as for existing images when a new vulnerability is added to the database. New in GA is support for more OSes; Container Registry vulnerability scanning is now available for Debian, Ubuntu, Red Hat Enterprise Linux, CentOS, and Alpine images.Binary Authorization is a deploy-time security control that ensures only trusted container images are deployed on Kubernetes Engine. Binary Authorization lets you define requirements for deployment—such as signed images and required scanning—so that only verified images are integrated into the build-and-release process. With the GA announcement, Binary Authorization introduces three new features:Global policy for GKE system containers. In order to use GKE, you need to run a number of GKE system containers in your environment. In the past, you had to manually ensure that only up-to-date and authentic system containers are deployed. With the GA of Binary Authorization, you can opt to only allow trusted system containers that are built and recognized by the GKE team to be deployed, gaining more control and visibility over your production environments.Dry-run, which allows customers to set a deploy policy in non-enforcement mode and use auditing to record and review any would-be blocked deployment. This gives you the flexibility to test out new policies without risking an interruption in your production release cycle.Support for KMS asymmetric keys (beta), allowing you to sign and verify container images using asymmetric keys generated by Cloud KMS, as well as support for generic PKCS signing keys in case you want to use your own PKI.Secure to run, thanks to isolation and early warningsYou can’t always control the contents of your workloads or be completely selective about what ends up in your environment. For example, you might be getting containers from your customers or third parties. When you run untrusted code, particularly in a multi-tenant environment, you want to be able to trust that the boundaries you have between your workloads is strong. The soon-to-be beta of GKE Sandbox brings Google’s gVisor sandboxing technology natively to GKE, using the Runtime Class. This provides you with a second layer of defense between containerized workloads, without changes to your applications, new architectural models, or added complexity. Container escape—when a compromised container gains access to the host and data in other containers—is a concern for anyone running sensitive workloads in containers. GKE Sandbox reduces the need for the container to interact directly with the host, shrinking the attack surface for host compromise, and restricting the movement of malicious actors.Whether you’re running your own (trusted) containers, or untrusted containers, however, you’ll want to lock down your Kubernetes configuration. We keep our hardening guide up to date,  leading you through the best practices to follow. At Next, we also introduced a simpler way with Security Health Analytics (alpha), which provides automated security checks for common misconfigurations in Google Cloud, including those discussed in the GKE hardening guide. The results of these checks are reported in the Cloud Security Command Center (Cloud SCC), so you have a single place to look at security reports for your clusters. (Don’t forget that we also have many partners who offer container runtime security, and who are directly integrated with the Cloud SCC!)Product announcements aside, Next ‘19 was also a great place to learn about container security. In case you missed them, catch the recordings of all the best container security content:Who protects what? Shared security in GKESecure Software Supply Chains on Google Kubernetes EngineSecuring Kubernetes SecretsGKE Networking DifferentiatorsKeyless entry: Securely access GCP services from KubernetesEnd-to-End Security and Compliance for your Kubernetes Software Supply ChainSecure Policy Management for Anthos (Cloud Services Platform)GKE Sandbox for Multi-tenancy and SecurityNext ‘19 was a milestone in our efforts to improve container security. See you again next year!
Quelle: Google Cloud Platform

Connecting Global Azure Bootcampers with a cosmic chat app

Every year on the same day, the Global Azure Bootcamp brings together cloud enthusiasts who are eager to learn about Microsoft Azure. This year, the Bootcamp will be held on April 27, 2019 in more than 300 locations worldwide where local communities will welcome developers and IT professionals for a day of workshops, labs, and talks.

It will be a great experience for tens of thousands of Azure Bootcampers all around the world to simultaneously participate in this event. We decided to add a little “cosmic touch” to the Bootcamp by letting the attendees greet each other with a chat app that’s powered by Azure Cosmos DB. The chat app will be made available on April 27, 2019 by following a link displayed in each Bootcamp location!

A global database for a global event

If you think about it, this marriage just makes sense. It’s a planet-scale bootcamp, and what better technology than a globally distributed database to serve a global event?

From the beginning, we tackled this project with the goal of a great user experience. For a chat app, this means low latency in the ingestion and delivery of messages. To achieve that, we deployed our web chat over several Azure regions worldwide and let Azure Traffic Manager route users’ requests to the nearest region where our Cosmos database was deployed to bring data close to the compute and the users being served. That was enough to yield near real-time message delivery performance as we let Azure Cosmos DB replicate new messages to each covered region.

To minimize ingestion latency, we also configured our Cosmos database to be deployed in multi-master mode, which makes it possible to write data to any region the database is deployed to.

The Azure Cosmos DB change feed as a publication system

You expect a chat app to display new messages in real-time, so these messages have to be pushed to the clients without the need for a slow and costly polling mechanism. We used Azure Cosmos DB’s change feed to subscribe to new messages being received and delivered them to each client through the Azure SignalR service.

We also chose to consume the change feed locally in each region to further minimize delivery latency, as having a single point of dispatch would have effectively reduced the benefits we got from being globally distributed and multi-mastered.

Here’s a high-level diagram showing the Azure services in play and how they interact:

Planet-scale, batteries included

So, what does it take to build such a distributed app? Nothing much really, as Azure Cosmos DB takes care of all the rocket science required! Since all the complexity involved with global data distribution is effectively hidden from the developer, what’s left are simple concepts that are easy to reason about:

You write messages to the region that’s closest to you.
You receive messages through the change feed, whatever region they were written from.

These concepts are straightforward to implement, and it only took around a day of development time to have the chat app up and running.

Keeping it friendly with automatic moderation

As our chat app was about to be made available for anyone to use, we also had to make sure to filter out any inappropriate message that may spoil the friendly spirit of the Global Azure Bootcamp. To achieve that, we used the content validation service powering the Content Moderator APIs from Azure Cognitive Services to scan each incoming message. This turnkey solution allows us to detect potential profanity or presence of personally identifiable information, and instantly block the related messages to ensure a positive and safe experience for everyone.

The database for real-time, global workloads

By ensuring the lowest latency in both the ingestion and delivery of messages, the combination of Azure Cosmos DB’s global footprint, multi-master capability, and change feed API let us achieve the best experience for anyone using the chat app, wherever they are on the planet.

Want to find out how Azure Cosmos DB’s unique characteristics can help you solve your next data challenges?

Head over to the Azure Cosmos DB website and learn more about the service.
Follow us on Twitter @AzureCosmosDB.
Ask your questions on StackOverflow with the tag [azure-cosmosdb].
Get started today using Azure Cosmos DB’s local emulator or try the service!

Quelle: Azure

Best practices in migrating SAP applications to Azure – part 2

This is the second blog in a three-part blog post series on best practices in migrating SAP to Azure.

Journey to SAP S/4HANA from SAP Business Suite

A common scenario where SAP customers can experience the speed and agility of the Azure platform is the ability to migrate from a SAP Business Suite running on-premises to SAP S/4HANA in the cloud. This scenario is a two-step process. The first step being migration from enterprise resource planning (ERP) on-premises to the Suite on HANA in Azure, and then converting from Suite on HANA to S/4HANA.

Using the cloud as the destination for such a migration project has the potential to save organizations millions of dollars in upfront cost for infrastructure equipment, and can shave roughly around 12 to 16 weeks off the project schedule as the most complex components of the infrastructure are already in place and can easily be provisioned when required. You can see where these time savings come from by looking at the time it takes to go through the request for proposal (RFP) process, to procure expensive servers with large memories, or potentially to procure dedicated appliances with only a five year lifespans such as storage arrays, network switches, backup appliances, and other data center enhancements. Not to mention the time and cost associated with an expertly trained team that can deploy S4/HANA based on tailored data center integration (TDI) standards as required by SAP. These are the same obstacles the consulting firm Alegri was running into during their experience migrating to S/4HANA on Azure. To read more about their experience, refer to Alegri's customer story, “Modern corporate management without any burden.”

While planning for a migration, you want to have a solid project plan that has all the tasks, milestones, dependencies, and stakeholders identified. You will also want a defined governance model and a RACI chart showing which group owns which tasks. Projects of this size have so many moving parts and dependencies between multiple internal and external teams. Disciple in governance is a must.

When getting started with the actual execution, you would typically want to identify all the SAP prerequisites specific to the versions that are currently in use. Larger SAP migration projects will typically have a number of parallel efforts going on at the same time. You want to be sure the infrastructure design on Azure is fully compliant with SAP’s support matrix. To do so, start by reviewing the SAP documentation and deployment guides on Azure,  Azure's SAP deployment checklist, and the product availability matrix (PAM). This is a great starting point because while you’re working on the initial landscape design, the basis admins can start running the readiness check looking at impact of the legacy custom code in the SAP Business Suite system, the S/4HANA sizing, and other S/4 perquisites required before the upgrade can begin. Together you’ll group on any new items that were uncovered before finalizing the design.

Next is the infrastructure deployment on Azure. The initial deployment is minimal, just enough infrastructure to support the initial sandbox and development tiers. Requirements at this state include:

An Azure subscription
An established VPN connection or Express Route
A deployed Azure network (VNets and Subnets) and security configurations (Network Security Groups)
The required number of virtual or bare metal servers and associated storage

The above requirements will comprise the new target environment. It is important to note that an ExpressRoute will be required to provide enough bandwidth for migrating medium to large databases. The basis admins can start installing the Suite on HANA database and the initial application servers. This will give the broader project team all the working area they need to run through as many migration runs as necessary to iron out any issues.

We get a lot of questions around the best migration method to use when migrating SAP from on-premises to Azure, which is understandable given the number of options including backup/restore, export/import, SUM and DMO, and more. When migrating the SAP Business Suite to S/4HANA on Azure, it’s easier to use the SUM and DMO option from SAP to handle the database migration and installing new application servers in Azure. This also provides an opportunity to have the new landscape in Azure on the latest supported operating system versions for optimal capabilities, performance, and support. My fellow Azure colleague Kiran Musunuru just recently authored a white paper on “Migration Methodologies for SAP on Azure” that outlines using the SUM and DMO option to migrate SAP Business Suite to S/4HANA as well as tools, options, individual steps, and the overall process of the migration.

Here are some tips that I’ve learned through my migration experience to help you avoid any last minute surprises.

Check the minimum required SAP versions and release levels across the landscape.
Check the PAM for Java components to see if they are required for your S/4HANA deployment.
Use the benchmark tool in DMO to get realistic migration times for the system. This helps with determining the best ExpressRoute sizing for the migration, and sets expectations for required downtimes.
Know the Unicode requirement to upgrade to S/4HANA.
Know the dual stack systems remediation.
Know that OS flavor and versions might differ from existing internal IT standards.
Plan for the Fiori deployment.
Be prepared to perform multiple migration mock runs for complex or very large landscapes.

Once the migration from the on-premises system to the Azure environment is complete, the conversion from the SAP Business Suite on HANA to S/4HANA can begin. Before starting I would review SAP’s conversion guide for the S/4HANA version that you will be upgrading to. For example, here is the conversion guide for S/4HANA 1809.

Deploying a new S/4HANA landscape can all be done in Azure by the basis team and eliminates any dependencies on the many internal IT teams that would be required with a deployment in a traditional on-premises environment. Most customers will likely start with a simple sandbox or new development system from a system copy of the newly migrated suite on the HANA system. This provides a staging area to run through all the simplification checks, custom code migrations, and actual S/4 conversion mock runs. In parallel, the basis team can start deploying the QAS and production tiers including the new VM’s and configuration for Fiori. Once the necessary checks and conversion have been completed, the process is simply repeated for QAS and production.

This blog post is meant as a general overview of the process and I didn’t touch on any of the functional areas as I wanted to focus on the items that are connected to the Azure platform. We also have a number of scripting and automation options around deploying S/4HANA landscapes on Azure. For an overview you can visit the blog post, “Automating SAP deployments in Microsoft Azure using Terraform and Ansible” written by Tobias Niekamp from the Azure Compute engineering team.

Next week, my colleague Troy Shane will cover the migration of SAP BW/4HANA. It will also cover how best to deploy SAP BW/4HANA in a scale-out architecture today and in the near future when our new Azure NetApp Files becomes generally available for SAP HANA workloads.
Quelle: Azure