da Agency

Navigating the intelligent edge: answers to top questions

Over the past ten years, Microsoft has seen embedded IoT devices get progressively smarter and more connected, running software intelligence near the point where the data is being generated within a network. And having memory and compute capabilities at the intelligent edge solves multiple conundrums related to connectivity, bandwidth, latencies, and privacy/security.

Of course, each device that connects to a network brings the challenge of how to secure, provision, and manage them. It raises issues of privacy requirements, data regulations, bandwidth, and transfer protocols. And when you have thousands of devices connecting to each other and broader systems like the cloud, all this can get very complex, very quickly.

Here are some of the most frequent questions around the intelligent edge and examples of how Azure solutions can help simplify securing, provisioning, and managing it. To hear more in-depth thoughts on this topic, join Olivier Bloch on October 10 as he speaks at the IoT in Action event in Santa Clara.
 

Securing the intelligent edge

“How do I ensure the devices that are connected are the ones they say they are, and that they are authenticating to the back end and securing data in an encrypted way?”

Each device that gets installed on a network provides one more potential network doorway for bad actors. No one wants their car radio, scale, or vending machine hacked. No one wants customer data stolen. We’ve already seen too much of that in the news. Securing the intelligent edge is rightfully a key concern for customers interested in IoT technology.

The key is to start simple by building on top of solutions that have addressed these important concerns. Microsoft intelligent edge and intelligent cloud solutions have been designed to complement each other, which makes it much easier to create secure IoT solutions that you can trust.

Azure Sphere is a great place to start. It provides a turnkey IoT solution that builds on decades of Microsoft experience, ensuring comprehensive, multi-layer security from the multipoint control unit (MCU) to the operating system to the cloud.

It begins with Azure Sphere-certified MCUs from our hardware partners, with Microsoft hardware root of trust embedded into the silicone. The operating system (OS) provides in-depth defense that guards against hackers and enables automated OS and security updates. The Azure Sphere Security Service safeguards every device with seven properties of highly secured, internet-connected devices. Azure Sphere only runs signed, authentic software, reducing risk of malware or application tampering. Even if you have devices that are already installed, they can be secured with Azure Sphere guardian modules, with little or no redesign required.

Provisioning and managing the intelligent edge

“Connecting one device manually to the cloud is part of the story. But what if I need to provision and then manage a whole bunch of devices at scale?”

You want to ensure devices are easy to provision, update, and manage. You want to be able to roll out new devices, and when the time comes, retire devices. You want to provision and manage devices like you would a fleet of PCs without having to manually update software and firmware.

Again, Microsoft has solutions that simplify all of this.

Azure IoT Hub enables you to connect, manage, and scale devices to the edge with per-device authentication and scaled provisioning. Azure IoT Edge, which is an intelligent edge runtime managed and configured from Azure IoT Hub, enables you to deploy cloud workloads to run on edge devices using standard containers. IoT Edge secures the communications between IoT applications and your edge devices, enabling you to power and remotely configure the devices. Built-in device management and provisioning capabilities enable you to connect and manage devices at scale.

To implement scaled provisioning, Azure IoT Hub is paired with the Device Provisioning Service (DPS) which streamlines the enrollment process by allowing you to register and provision all your devices to IoT Hub without any human intervention. DPS takes advantage of hardware-secured modules where secure seeds are planted by silicon manufacturers and confidential compute is possible, all to establish a trusted connection and authentication with a global endpoint (DPS). This, in turn, can be configured to not only provide IoT Hub device identity and credentials back to devices, but it also can deliver a first configuration at provisioning time. It’s a powerful and scalable way to manage IoT devices during their whole life cycle from the first connection to retirement, including transfers of ownership.

Learn more about the intelligent edge at an IoT in Action event

Microsoft continues to innovate with solutions that help streamline and simplify securing, provisioning, and managing the intelligent edge. To learn more about how you can best leverage this technology, be sure to register for the upcoming Santa Clara IoT in Action event on October 10. As part of the event, I will be leading a panel discussion focused on how customers and partners are simplifying IoT and solving industry problems. 

If you can’t make it to the Santa Clara event, there will also be one-day events held in cities around the world, including Warsaw, Frankfurt, Toronto, Auckland, Taipei, Shenzhen, and more. These events are a valuable opportunity to get all your questions answered and build connections with potential IoT partners. Through interactive sessions, Microsoft will share how various solutions and accelerators can help simplify IoT so you can get secure solutions out the door faster and more cost effectively.

Prefer a virtual event? Browse the IoT in Action webinar series which features IoT industry experts discussing real-life solution use cases. You can also get started on further advancing your technical IoT skills by watching the IoT Show, joining the IoT Tech Community, and learning at IoT School.
Quelle: Azure

da Agency

How to develop your service health alerting strategy

Service issues are anything that could affect your availability, from outages and planned maintenance to service transitions and retirements. While rare—and getting rarer all the time, thanks to innovations in impactless maintenance and disciplines like site reliability engineering—service issues do occur, which is why service health alerting is such a critical part of successfully managing cloud operations. It’s all about helping your team understand the status and health of your environment so you can act quickly in the event of an issue. That can mean taking corrective measures like failing over to another region to keep your app running or simply communicating with your stakeholders so they know what’s going on.

In this blog, we’ll cover how you can develop an effective service health alerting strategy and then make it real with Azure Service Health alerts.

How Azure Service Health alerts work

Azure Service Health is a free Azure service that provides alerts and guidance when Azure service issues like outages and planned maintenance affect you. Azure Service Health is available in the portal as a dashboard where you can check active, upcoming, and past issues.

Of course you may not want to check the Azure Service Health dashboard regularly. That’s why Azure Service Health also offers alerts. Azure Service Health alerts automatically notify you via your preferred channel such as email, SMS, mobile push notification, webhook into your internal ticketing system like ServiceNow or PagerDuty, and more if there’s an issue affecting you.

If you’re new to Azure Service Health alerts, you’ll notice that there are many choices to make during the configuration process. Who should I alert about which services and regions? Who should I alert for which types of health events? Outages? Planned maintenance? Health advisories? And what type of notification like email, SMS, push notification, webhook, or something else should I use?

To answer these questions the right way, you’ll need to have a conversation with your team and develop your service health alerting strategy.

How to develop your service health alerting strategy with your team

There are three key considerations for your team to address when you set up your Azure Service Health alerts.

First, think about criticality. How important is a given subscription, service, or region? If it’s production, you’ll want to set up an alert for it, but dev/testing might be unnecessary. Azure Service Health is personalized, so we won’t trigger your alert if the service issue affects a service or region you aren’t using.

Next, decide who to inform in the event of an issue. Who is the right person or team to tell about a service issue so they can act? For example, send Azure SQL or Azure Cosmos DB issues to your database team.

Finally, agree on how to inform that individual or team. What is the right communication channel for the message? Email is noisy, so it might take longer for your teams to respond. That’s fine for planned maintenance that’s weeks away, but not for an outage affecting you right now, in which case you’ll want to alert your on-call team using a channel that’s immediately seen, like a push notification or SMS. Or if you’re a larger or more mature organization, plug the alerts into your existing problem management system using a webhook/ITSM connection so you can follow your normal workflow.

For more information on Azure Service Health, how to set up alerts, and other critical guidance for handling service issues including, in some cases, avoiding their impact altogether, check out the video below:

Set up your Azure Service Health alerts today

Once you’ve had your Azure Service Health alerting conversation with your team and developed your strategy, configure your Azure Service Health alerts in the Azure Portal.

For more in-depth guidance, visit the Azure Service Health documentation. Let us know if you have a suggestion by submitting an idea via our feedback forum.
Quelle: Azure

da Agency

Google Cloud named a leader in the Forrester Wave: Streaming Analytics

We’re pleased to announce that Forrester has named Google Cloud as a leader in The Forrester Wave™: Streaming Analytics, Q3 2019. We believe the findings reflect Google Cloud’s market momentum, and what we hear from our satisfied enterprise customers who are using Cloud Pub/Sub and Cloud Dataflow in production for streaming analytics in conjunction with our broader platform.  According to Forrester, many leading enterprises realize that real-time analytics—the analytics of what’s happening with data in the present—is an incredible competitive advantage. With it, they can act right away to serve customers, fix operational problems, power internet of things (IoT) apps, and respond decisively to competitors. The report evaluates the top 11 vendors against 26 rigorous criteria for streaming analytics to help enterprise IT teams understand their options and make informed choices for their organizations. Google scored 5 out of 5 in Forrester’s report evaluation criteria of scalability, availability, aggregates, management, security, extensibility, ability to execute, solution roadmap, partners, community, and customer adoption. How Cloud Pub/Sub and Cloud Dataflow work for usersThere’s a lot of pressure on companies today to become data-driven, but they need high-performing technology to make that a reality. As part of our larger cloud data analytics platform, Cloud Pub/Sub and Cloud Dataflow are designed for ease of use, scalability, and performance. We’re especially pleased that our recognition as a Leader in the Forrester Wave: Streaming Analytics mirrors what we hear from our customers: The report notes that “Google Cloud unifies streaming analytics and batch processing the way it should be. No compromises.” While stream analytics is a leading business priority, real-life use cases frequently require batch data as an input as well. Google Cloud Platform (GCP) customers can simplify their pipeline development by reusing code across both batch and stream processing using Apache Beam, which also provides pipeline portability to OSS projects. Further, Google Cloud’s data analytics portfolio autoscales across ingestion, processing, and analysis, which eliminates the need for provisioning and makes handling varying volumes of streaming data automatic. We hear from users that they’re able to ingest and process data much more quickly than in the past, helping to get new business insights faster and allowing more users to do self-serve analytics.Cloud Pub/Sub is our stream event ingestion service, which uses a publish/subscribe pattern to deliver events across stream, batch, and unified pipelines. Cloud Pub/Sub is a global service that enables data engineers to automatically scale without provisioning, partitioning, or isolation concerns. We hear that users choose this because it allows them to be production-ready from day one, with end-to-end encryption, virtually unlimited throughput, and high data durability and availability with cross-zone replication. They’re able to access more data faster to find new insights and focus on new projects, not managing infrastructure.We are very excited about the productivity benefits offered by Cloud Dataflow and Cloud Pub/Sub. It took half a day to rewrite something that had previously taken over six months to build using Apache Spark. Paul Clarke, Director of Technology, OcadoCloud Dataflow offers simplified stream and batch processing in one service without compromises, which the Forrester report says “must be the goal when software architects create a unified streaming and batch solution that must scale elastically, perform complex operations, and have the resiliency of Rocky Balboa. Google’s Cloud Dataflow is that solution for Google Cloud Platform.” In addition to scaling automatically, Cloud Dataflow is deeply integrated with our other GCP services (including Cloud Pub/Sub, in particular), provides exactly-once processing with built-in fault tolerance, and automates performance, availability, security, and compliance.Cloud Dataflow’s streaming capabilities allow businesses to capture fleeting opportunities by analyzing real-time events while the window to act on analysis is still open. “Google Cloud Dataflow allowed us to rapidly prototype, iterate, and launch a next-generation platform for analytics at Twitter. It has been able to easily scale to our needs, handling multiple TB of streaming and batch data per hour,” says Steve Niemitz, senior software engineer, Twitter. “Being fully managed by Google allows us to focus on value-add for our users rather than operational issues. Additionally, the ability to write a single job in Beam and then run it in both streaming and batch mode has drastically simplified our codebase and operations.”We’re pleased with our ranking as a Leader in this Wave. However, as the 5/5 evaluation score in the solution roadmap criterion indicates, what we’re most excited about is the opportunity to continue to develop our stream analytics solution to be even more powerful for our customers. Stay tuned as we continue developing new capabilities, expanding the number of ways our customers can drive real value from real-time analytics. Download the full Forrester report, and learn more about GCP streaming analytics here.
Quelle: Google Cloud Platform

da Agency

Designing Your First App in Kubernetes, Part 1: Getting Started

Image credit: Evan Lovely

Kubernetes: Always Powerful, Occasionally Unwieldy
Kubernetes’s gravity as the container orchestrator of choice continues to grow, and for good reason: It has the broadest capabilities of any container orchestrator available today. But all that power comes with a price; jumping into the cockpit of a state-of-the-art jet puts a lot of power under you, but how to actually fly the thing is not obvious. 
Kubernetes’ complexity is overwhelming for a lot of people jumping in for the first time. In this blog series, I’m going to walk you through the basics of architecting an application for Kubernetes, with a tactical focus on the actual Kubernetes objects you’re going to need. I’m not, however, going to spend much time reviewing 12-factor design principles and microservice architecture; there are some excellent ideas in those sort of strategic discussions with which anyone designing an application should be familiar, but here on the Docker Training Team I like to keep the focus on concrete, hands-on-keyboard implementation as much as possible.
Furthermore, while my focus is on application architecture, I would strongly encourage devops engineers and developers building to Kubernetes to follow along, in addition to readers in application architecture roles. As container orchestration becomes mainstream, devops teams will need to anticipate the architectural patterns that application architects will need them to support, while developers need to be aware of the orchestrator features that directly affect application logic, especially around networking and configuration consumption. 
Just Enough Kube
When starting out with a machine as rich as Kubernetes, I like to identify the absolute minimum set of things we’ll need to understand in order to be successful; there’ll be time to learn about all the other bells and whistles another day, after we master the core ideas. No matter where your application runs, in Kubernetes or anywhere else, there are four concerns we are going to have to address:

 Processes: Your actual running code, compiled or interpreted, is the core of your application. We’re going to need a set of tools not only to schedule these processes, but to maintain and scale those processes over time. For this, we’re going to use pods and controllers.
 Networking: The processes that make up your application will likely need to talk to each other, external resources, and the outside world. We’re going to need tooling to allow us to do service discovery, load balancing and routing between all the components of our application. For this, we’re going to use Kubernetes services.
 Configuration: A well-written application factors out its configuration, rather than hard-coding it. This is a direct consequence of applying the paradigm of Don’t Repeat Yourself when coding; things that may change based on context, like access tokens, external resource locations, and environment variables should be defined in exactly one place which can be both read from and updated as needed. An orchestrator should be able to provision configuration in modular fashion, and for this we’re going to use volumes and configMaps.
 Storage: Well-built applications always assume their containers will be short lived, and that their filesystems will be destroyed with no warning. Any data collected or generated by a container, as well as any data that needs to be provisioned to a container, should be offloaded to some sort of external storage. For this, we’ll look at Container Storage Interface plugins and persistentVolumes.

And that’s it. I’ll provide some ‘advanced topics’ pointers throughout the blog series to give you some ideas on what to study after you’ve mastered the basics in order to take your Kubernetes apps even further. When you’re starting out, focus on the components mentioned above and detailed in this series.
Just Enough High-Level Design
I promised above to keep this series more tactical than strategic, but there are some high-level design points we absolutely need to understand the engineering decisions that follow, and to make sure we’re getting the maximum benefit out of our containerization platform. Regardless of what orchestrator we’re using, there are three key principles we need to keep in mind that set a standard for what we’re trying to achieve when containerizing applications: portability, scalability, and shareability:

 Portability: Whatever we build, we should be able to deploy it on any Kubernetes cluster; this means not having hard dependencies on any feature or configuration of the underlying host or its filesystem. If the idea of moving your app from your dev machine to a testing server sounds stressful, something probably needs to be rethought.
 Scalability: Containerized applications scale best when they scale horizontally: by adding more containers, and not just containers with more compute resources. It doesn’t matter how many resources are allocated to a container; they are still mortal and often short-lived objects managed by your orchestrator as it tries to adapt to changing cluster conditions and load. Therefore, we’re going to need to arrange our applications to easily leverage more copies of the containers it expects, typically by using the routing and load balancing features of our orchestrator, and by trying to make our containers stateless whenever possible.
 Shareability: We don’t want to be trapped maintaining and consulting on every app we build forever. It’s crucial that we’re able to share our apps with other developers who we may hand it off to in future, operators who have to manage it in production, and third parties who may be able to leverage it in an open-source context. We’re halfway there with portability, ensuring that it’s possible to move our app from cluster to cluster, but beyond it just being technically possible, shareability emphasizes that hand off should also be easy and reliable. Standing up our app on a new cluster should be as foolproof as possible, at least for a first pass.

Thinking Through Your First Application on Kubernetes
For the rest of this series, let’s think through containerizing a simple three-tier web app for Kubernetes, with the following typical components:

 A database for holding all the data required by the application
 An API which is allowed to access the database
 A frontend which is reachable by users on the web, and which uses the API to interact with the database

Even if applications like these are the furthest thing from what you work with, the example is instructive; we’ll focus on decision points that are widely applicable to many different kinds of applications, so you can see examples of how to make these decisions. The generic application above is just a vehicle for touring the relevant concepts, but they apply quite generally.
Let’s begin by imagining that you’ve already created Docker images for each component of your application, whether they resemble the components listed above or are completely different. If you’d like a primer on designing and building Docker images, see my colleague Tibor Vass’s excellent blog post on dockerfile best practices.
Checkpoint #1: Make your images.
Before you’ll be able to orchestrate anything, you’ll need images built for every type of container you want to run in your application.
Also note, we’re going to consider some of the simplest cases for each concern; start with these, and when you master them, see the Advanced Topics subsection in each step for pointers to what topics to explore next.
You’ve made it this far! In the next post, I’ll explore setting up processes as pods and controllers.
We will also be offering training on Kubernetes starting in early 2020. To get notified when the training is available, sign up here:
Get Notified About Training
To learn more about running Kubernetes with Docker:

Try the Docker Kubernetes Service, the easiest way to securely run and manage Kubernetes in the enterprise.
Try out Play with Kubernetes.
Find out how to simplify Kubernetes with Docker Compose.

Getting Started: How To Build Your First Application in #Kubernetes (part 1)Click To Tweet

The post Designing Your First App in Kubernetes, Part 1: Getting Started appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

da Agency

Building a document understanding pipeline with Google Cloud

Document understanding is the practice of using AI and machine learning to extract data and insights from text and paper sources such as emails, PDFs, scanned documents, and more.In the past, capturing this unstructured or “dark data” has been an expensive, time-consuming, and error-prone process requiring manual data entry. Today, AI and machine learning have made great advances towards automating this process, enabling businesses to derive insights from and take advantage of this data that had been previously untapped. In a nutshell, document understanding allows you to:Organize documents Extract knowledge from documents Increase processing speedAt Google Cloud we provide a solution, Document AI, which enterprises can leverage in collaboration with partners to implement document understanding. However, many developers have both the desire and the technical expertise to build their own document understanding pipelines on Google Cloud Platform (GCP)—without working with a partner—using the individual Document AI products.If that sounds like you, this post will take you step-by-step through a complete document understanding pipeline. The Overview section explains how the pipeline works, and the step-by-step directions below walk you through running the code.OverviewIn order to automate an entire document understanding process, multiple machine learning models need to be trained and then daisy-chained together alongside processing steps into an end-to-end pipeline. This can be a daunting process, so we have provided sample code for a complete document understanding system mirroring a data entry workflow capturing structured data from documents.Our example end-to-end document understanding pipeline consists of two components:A training pipeline which formats the training data and uses AutoML to buildImage Classification,Entity Extraction,Text Classification, andObject Detection models.A prediction pipeline which takes PDF documents from a specified Cloud Storage Bucket, uses the AutoML models to extract the relevant data from the documents, and stores the extracted data in BigQuery for further analysis.Training DataThe training data for this example pipeline is from a public dataset containing PDFs of U.S. and European patent title pages with a corresponding BigQuery table of manually entered data from the title pages. The dataset is hosted by the Google Public Datasets Project.Part 1: The Training Pipeline The training pipeline consists of the following steps:Training data is pulled from the BigQuery public dataset. The training BigQuery table includes links to PDF files in Google Cloud Storage of patents from the United States and European Union.The PDF files are converted to PNG files and uploaded to a new Cloud Storage bucket in your own project. The PNG files will be used to train the AutoML Vision models.The PNG files are run through the Cloud Vision API to create TXT files containing the raw text from the converted PDFs. These TXT files are used to train the AutoML Natural Language models.The links to the PNG or TXT files are combined with the labels and features from the BigQuery table into a CSV file in the training data format required by AutoML. This CSV is then uploaded to a Cloud Storage bucket. Note: This format is different for each type of AutoML model.This CSV is used to create an AutoML dataset and model. Both are named in the format patent_demo_data_%m%d%Y_%H%M%S. Note that some AutoML models can sometimes take hours to train.Part 2: The Prediction PipelineThis pipeline uses the AutoML models previously trained by the pipeline above. For predictions, the following steps occur:The patent PDFs are collected from the prescribed bucket and converted to PNG and TXT files with the Cloud Vision API (just as in the training pipeline).The AutoML Image Classification model is called on the PNG files to classify each patent as either a US or EU patent. The results are uploaded to a BigQuery table.The AutoML Object Detection model is called on the PNG files to determine the location of any figures on the patent document. The resulting relative x, y coordinates of the bounding box are then uploaded to a BigQuery table.The AutoML Text Classification model is called on the TXT files to classify the topic of the patent content as medical technology, computer vision, cryptocurrency or other. The results are then uploaded to a BigQuery table.The AutoML Entity Extraction model is called to extract predetermined entities from the patent. The extracted entities are applicant, application number, international classification, filing date, inventor, number, publication date and title. These entities are then uploaded to a BigQuery table.Finally, the BigQuery tables above are joined to produce a final results table with all the properties above.Step-by-Step DirectionsFor the developers out there, here’s how you can build the document understanding pipeline of your dreams. You can find all the code in our GitHub Repository.Before you begin:You’ll need a Google Cloud project to run this demo. We recommend creating a new project.We recommend running these instructions in Google Cloud Shell(Quickstart). Other environments will work but you may have to debug issues specific to your environment. A Compute Engine VM would also be a suitable environment.1. Git clone the repo and navigate to the patents example.2. Install the necessary system dependencies. Note that in Cloud Shell system-wide changes do not persist between sessions, so if you step away while working through these step-by-step instructions you will need to rerun these commands after restarting Cloud Shell.3. Create a virtual environment and activate it.When your virtual environment is active you’ll see patents-demo-env in your command prompt. Note: If your Cloud Shell session ends, you’ll need to reactivate the virtual environment by running the second command again.For the remaining steps, make sure you’re in the correct directory in the professional-services repo: /examples/cloudml-document-ai-patents/.4. Install the necessary libraries into the virtual environment.5. Activate the necessary APIs.Note: We use the gcloud SDK to interact with various GCP services. If you are not working in Cloud Shell you’ll have to set your GCP project and authenticate by running gcloud init.6. Edit the config.yaml file. Look for the configuration parameter project_id (under pipeline_project) and set the value to the project id where you want to build and run the pipeline. Make sure to enclose the value in single quotes, e.g. project_id: ‘my-cool-project’.Note: if you’re not used to working in a shell, run nano to open a simple text editor.7. Also in config.yaml, look for the configuration parameter creator_user_id (under service_acct) and set the value to the email account you use to log in to Google Cloud. Make sure to enclose the value in single quotes, e.g. creator_user_id: ‘cloud.user@fake.email.address.com’.8. Create and download a service account key with the necessary permissions to be used by the training and prediction pipelines.9. Run the training pipeline. This may take 3–4 hours, though some models will finish more quickly.Note: If Cloud Shell Closes while the script is still downloading, converting, or uploading the PDFs, you will need to reactivate the virtual environment, navigate to the directory, and rerun the pipeline script. The image processing should take about 15-20 minutes, so make sure Cloud Shell doesn’t close during that time.10. After training the models (wait about 4 hours), your Cloud Shell session has probably disconnected. Reconnect to Cloud Shell and run the following commands to reinstall dependencies, reactivate the environment, and navigate to the document understanding example code.11. Next, you need to use the AutoML UIs to deploy the object detection and entity extraction models, and to find the ids of your models (which you will enter into config.yaml). In the UIs, you can also view relevant evaluation metrics about the model and see explicit examples where your model got it right (and wrong).Note: Some AutoML products are currently in beta; the look and function of the UIs may change in the future.Go to the AutoML Image Classification models UI, and make sure you are in same project you ran the training pipeline in (top right dropdown). Note the ID of your trained image classification model. Also note the green check mark to the right of the model—if this is not a check mark, or if there is nothing listed, it means model training is still in progress and you need to wait.In Cloud Shell, edit config.yaml. Look for the line model_imgclassifier: and on the line below you’ll see model_id:. Put the image classification model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890’Go to the AutoML Natural Language models UI, and similarly, make sure you are in the correct project, make sure your model is trained (green check mark) and note the model id of the trained text classification model.In Cloud Shell, edit config.yaml. Look for the line model_textclassifier: and on the line below you’ll see model_id:. Put the text classification model id after model_id: in single quotes, so the line looks something like ” model_id: ‘ABC1234567890′”Go to the AutoML Object Detection models UI, again making sure your project is correct and your model is trained. In this UI, the model id is below the model name. You also need to deploy the model, by opening the menu under the three dots at the far right of the model entry and selecting “Deploy model”. A UI popup will ask how many nodes to deploy to, 1 Node is fine for this demo. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the deployment status is in the UI under “Deployed” and it will be “Yes” when the model is deployed.In Cloud Shell, edit config.yaml. Look for the line model_objdetect: and on the line below you’ll see model_id:. Put the object detection model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890’Go to the AutoML Entity Extraction models UI, and make sure the project and region are correct. Verify the model training is complete by checking to make sure the Precision and Recall metrics are listed. Note the id as well.To deploy the model, click the model name and a new UI view will open. Click “DEPLOY MODEL” near the top right and confirm. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the UI will update when deployment is complete.In Cloud Shell, edit config.yaml. Look for the line model_ner: and on the line below you’ll see model_id:. Put the entity extraction model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890′.Before going further, make sure the model deployments are complete.12. To run the prediction pipeline, you’ll need some PDFs of patent first pages in a folder in Cloud Storage. You can provide your own PDFs in your own Cloud Storage location, or for demonstration purposes, you can run the following commands, which create a bucket with the same name as your project id (if it doesn’t already exist) and copy a small set of five patent first pages into a folder in the bucket called patent_sample.13. In the config.yaml file, fill in the configuration parameter labeled demo_sample_data (under pipeline_project) with the Cloud Storage location of the patents you want to process (in single quotes). If you’re following the example above, the parameter value is ‘gs://<YOUR-PROJECT-ID>/patent_sample’, substituting your project id.Also, fill in the parameter labeled demo_dataset_id, just under demo_sample_data. With the BigQuery dataset (in single quotes) in your project where the predicted entities will be collected. For example, you can use ‘patent_demo’. Note that this dataset must not exist already; the code will attempt to create the dataset and will stop running if the dataset already exists.Now, you are ready to make predictions!14. Run the prediction pipeline. This will process all the patent PDF first pages in the Cloud Storage folder specified in the demo_sample_data parameter, and upload predictions to (and create) BigQuery tables in the dataset specified by the  demo_dataset_id parameter.python3 run_predict.py15. Finally, go check out the dataset in the BigQuery UI and examine all the tables that have been created. There is a table called “final_view” which collects all the results in a single table. You should see something like this:Note, for this, or any of the other intermediate tables that were created during the prediction pipeline, you may need to query the table to see the results since it is so new. For example:If you are new to BigQuery, you can open a querying interface in the UI prepopulated with the table name by clicking the “QUERY TABLE” button visible when you select a table to view.ConclusionCongratulations! You now have a fully functional document understanding pipeline that can be modified for use with any PDF documents. To modify this example to work with your own documents, the data collection and training stages will need to be modified to pull documents from a local machine or another Cloud Storage bucket rather than a public BigQuery dataset. The training data will also need to be created manually for the specific type of documents you will be using. Now, if only we could create a robot that could scan paper documents to PDFs…For more information about the general process of document understanding, see this blog post from Nitin Aggarwal at Google Cloud. And for more information about the business use cases of Document Understanding on Google Cloud, check out this session from Google Cloud Next ’19.
Quelle: Google Cloud Platform

da Agency

Cloud Build named a Leader for Continuous Integration in the Forrester Wave

Today, we are honored to share that Cloud Build, Google Cloud’s continuous integration (CI) and continuous delivery (CD) platform, was named a Leader in The Forrester Wave™: Cloud-Native Continuous Integration Tools, Q3 2019. The report identifies the 10 CI providers that matter most for continuous integration (CI) and how they stack up on 27 criterias. Cloud Build received the highest score in both the current offering and strategy categories. “Google Cloud Build comes out swinging, matching up well with other cloud giants. Google Cloud Build is relatively new when compared to the other public cloud CI offerings; this vendor had a lot to prove, and it did…. Customer references are happy to trade the cost of paying for the operation and management of build servers for moving their operations to Google’s pay-per-compute system. One reference explained that it’s in the middle of a cloud shift and is currently executing up to 650 builds per day. With that proved out, this customer plans to move an additional 250 repositories to the cloud” – Forrester Wave™ reportTop score in the Current Offering category: Among all 10 CI providers evaluated in the Wave, Cloud Build got the highest score in the Current Offering category. The Current Offering score is based on Cloud Build’s strength in the developer experience, build speed and scale, enterprise security and compliance, and enterprise support, amongst other criteria.Top score in the Strategy category: Along with top score in the current offering category, Cloud Build also received the highest score in the Strategy category. In particular, Cloud Build’s scores in the partner ecosystem, commercial model, enterprise strategy and vision, and product roadmap criteria contributed to the score. CI plays an increasingly important role in DevOps, allowing enterprises to drive quality from the start of their development cycle. The report mentions “that cloud-native CI is the secret development sauce that enterprises need to be fast, responsive, and ready to take on incumbents and would-be digital disruptors.”At Google, we’ve seen first-hand how cloud-based serverless CI tool can help drive quality and security at scale, and the lessons we’ve learned in that process manifest directly in Cloud Build. Today, organizations of all sizes use Cloud Build to drive productivity improvements via automated, repeatable CI processes. Some customers include Zendesk, Shopify, Snap, Lyft, and Vendasta, who chose Cloud Build for its:Fully serverless platform: Cloud Build scales up and scale down in response to load with no need to pre-provision servers or pay in advance for additional capacity. Flexibility: With custom build steps and pre-created extensions to third party apps, enterprises can easily tie their legacy or home-grown tools as a part of their build process.Security and compliance features: Developers have an ability to perform deep security scans within the CI/CD pipeline and ensure only trusted container images are deployed to production. We’re thrilled by Forrester’s recognition for Cloud Build. You can download a copy of the report here.
Quelle: Google Cloud Platform

da Agency

Deployed AI: How Lumiata is using AI to make healthcare smarter

Editor’s note: Today we hear from Miguel Alvarado, CTO at Lumiata, which is using Google Cloud AI to provide intelligent health analytics for improved risk awareness and cost management in healthcare. Lumiata was founded on a vision of smarter, more cost-effective healthcare, and AI plays a central role in making it a reality. We provide sophisticated machine learning applications that elevate a healthcare organization’s ability to manage spend forecasting, stop loss, disease prediction and more, out of the box. We recently unveiled the Lumiata AI Platform for healthcare organizations who are seeking to apply machine learning to improve affordability and quality of care. Our platform helps healthcare organizations in two ways. Firstly, they can streamline their data to get a comprehensive longitudinal person record (LPR) for personalized analytics that uses different signals including claims, eligibility, labs, pharmacy, EHR/EMR, unstructured data, and more. Secondly, they get tools and access to pre-trained models through Lumiata AI Studio that healthcare data scientists can use to build and deploy machine learning models for use-cases like predicting medical costs and events.It takes powerful infrastructure to deliver ease of use and simplicity to our customers. After careful evaluation, we chose Google Cloud to help us deliver meaningful AI capabilities to transform healthcare organizations. Google Cloud’s security infrastructure, wide selection of intuitive AI tools, and technologies like Anthos that make multi-cloud environments utterly straightforward were some of the key factors behind our decision.As we moved to Google Cloud and built our platform, there are a few lessons we learned along the way that can help others who are in the middle of deploying AI in business. Lesson #1: AI doesn’t “just work”The first misconception about AI—and probably the most common—is that it’s a kind of silver bullet that “just works”. The truth is, AI must be purpose-built to extract the most value and insights from data. It requires a deep understanding of the problem at hand and the unique challenges it presents. With a goal to improve the affordability and quality of care, we focus on helping healthcare organizations use AI to understand the individual and provide personalized experiences. Google addresses this learning head-on with Deployed AI, a uniquely pragmatic approach that echoes our belief that although AI is incredibly powerful—and often transformative—it’s not magic.It also takes patience to iterate, experiment, and learn from mistakes, from stakeholders at every level of an organization. Operating ML infrastructure and live production models is hard.The CI/CD philosophy you apply to traditional software can be applied to models as well. These practices—also called “MLOps”—may not be the most attractive parts of AI, but they’re an essential part of successful deployments. We’re doing the ML Ops heavy lifting on behalf of our customers. Through Lumiata’s MLOps framework, models can be automatically monitored, validated, and tested so model performance is continuously optimized.Lesson #2: AI is about much more than modelsNext, I believe a disproportionate amount of attention is paid to machine learning models themselves, making it easy to underestimate the universe of supporting components that surrounds them and makes them work. There’s the data that trains the model, for instance, as well as the infrastructure that connects everything—two areas in which Google has deeper roots than just about anyone. Healthcare organizations are sitting on large volumes of data, but its visibility and access to the data science and analytics teams are limited. This impedes their ability to apply AI to meaningfully solve relevant problems. BigQuery, Google’s serverless data warehouse, enables us to solve this challenge for our users by providing access to large volumes of data with no operational hurdle at all.In addition to data, building, deploying, and maintaining machine learning models requires significant computational horsepower. Our customers often struggle to get buy-in from their technology counterparts to scale their infrastructure requirements for machine learning. With the help of Google Kubernetes Engine, our platform helps them meet their compute requirements for machine learning with ease and flexibility. Lesson #3: AI is a team sportThe bottom line is that even the most sophisticated machine learning model is but a single part of a larger system. Building that system—an AI deployment that makes a real-world difference—takes software engineering muscle, a culture of DevOps, and the tooling developers need to rapidly automate, test, deploy, and measure the results.Smaller companies often benefit from generalists that wear a number of hats, while bigger companies can afford to hire a range of specialists. Either way, engagement between disciplines is key: for instance, it’s vital to encourage machine learning experts to participate in development activities like testing and automation. There’s also the question of proportion; both engineering talent and AI vision are vital, but the former is generally needed in larger numbers than the latter. There’s no substitute for at least one qualified voice to push the envelope with advanced models that optimize for multiple variables or deliver transparency reasoning—sometimes known as “explainability”—along with their predictions; but it takes a far larger group of engineers to bring such a vision into production in a robust, efficient way.Adoption of AI in healthcare is limited partially due to the talent gap. Recruiting and retaining top talent to build an AI-architecture can be challenging. We strive to alleviate this challenge for our customers through our platform by automating many aspects of the machine learning pipeline, and thereby enabling them to deploy AI faster.  Finally, be ready to continually test your deployment once it goes into production. Real-time monitoring helps ensuring models perform as expected and remain consistent. ConclusionAI may be complex, but we remain big believers in its remarkable potential to transform business. That’s why we founded Lumiata—to address these challenges ourselves so healthcare organizations can focus on innovations that improve affordability and quality of care.We’re helping to make healthcare smarter. And with Google Cloud, we’re getting there faster than ever.Learn more about AI on Google Cloud.
Quelle: Google Cloud Platform