Kategorie: Microsoft Azure

Automated machine learning and MLOps with Azure Machine Learning

Azure Machine Learning is the center for all things machine learning on Azure, be it creating new models, deploying models, managing a model repository, or automating the entire CI/CD pipeline for machine learning. We recently made some amazing announcements on Azure Machine Learning, and in this post, I’m taking a closer look at two of the most compelling capabilities that your business should consider while choosing the machine learning platform.

Before we get to the capabilities, let’s get to know the basics of Azure Machine Learning.

What is Azure Machine Learning?

Azure Machine Learning is a managed collection of cloud services, relevant to machine learning, offered in the form of a workspace and a software development kit (SDK). It is designed to improve the productivity of:

Data scientists who build, train and deploy machine learning models at scale
ML engineers who manage, track and automate the machine learning pipelines

Azure Machine Learning comprises of the following components:

An SDK that plugs into any Python-based IDE, notebook or CLI
A compute environment that offers both scale up and scale out capabilities with the flexibility of auto-scaling and the agility of CPU or GPU based infrastructure for training
A centralized model registry to help keep track of models and experiments, irrespective of where and how they are created
Managed container service integrations with Azure Container Instance, Azure Kubernetes Service and Azure IoT Hub for containerized deployment of models to the cloud and the IoT edge
Monitoring service that helps tracks metrics from models that are registered and deployed via Machine Learning

Let us introduce you to Machine Learning with the help of this video where Chris Lauren from the Azure Machine Learning team showcases and demonstrates it.

As you see in the video, Azure Machine Learning can cater to workloads of any scale and complexity. Please see below, a flow for the connected car application demonstrated in the video. This is also a canonical pattern for machine learning solutions built on Machine Learning:

Visual: Connected Car demo architecture leveraging Azure Machine Learning

Now that you understand Azure Machine Learning, let’s look at the two capabilities that stand out:

Automated machine learning

Data scientists spend an inordinate amount of time iterating over models during the experimentation phase. The whole process of trying out different algorithms and hyperparameter combinations until an acceptable model is built is extremely taxing for data scientists, due to the monotonous and non-challenging nature of work. While this is an exercise that yields massive gains in terms of the model efficacy, it sometimes costs too much in terms of time and resources and thus may have a negative return on investment (ROI).

This is where automated machine learning (ML) comes in. It leverages the concepts from the research paper on Probabilistic Matrix Factorization and implements an automated pipeline of trying out intelligently-selected algorithms and hypermeter settings, based on the heuristics of the data presented, keeping into consideration the given problem or scenario. The result of this pipeline is a set of models that are best suited for the given problem and dataset.

Visual: Automated machine learning

 

Automated ML supports classification, regression, and forecasting and it includes features such as handling missing values, early termination by a stopping metric, blacklisting algorithms you don’t want to explore, and many more to optimize the time and resources.

Automated ML is designed to help professional data scientists be more productive and spend their precious time concentrating on specialized tasks such as tuning and optimizing the models, alongside mapping real-world cases to ML problems, rather than spending time in monotonous tasks like trial and error with a bunch of algorithms. Automated ML with its newly introduced UI mode (akin to a wizard) also helps open the doors of machine learning to novice or non-professional data scientists as they can now become valuable contributors in data science teams by leveraging these augmented capabilities and churning out accurate models to accelerate time to market. This ability to expand data science teams beyond the handful of highly specialized data scientists enables enterprises to invest and reap the benefits of machine learning at scale without having to compromise high-value use cases due to the lack of data science talent.

To learn more about automated ML in Azure Machine Learning, explore this automated machine learning article.

Machine learning operations (MLOps)

Creating a model is just one part of an ML pipeline, arguably the easier part. To take this model to production and reap benefits of the data science model is a completely different ball game. One has to be able to package the models, deploy the models, track and monitor these models in various deployment targets, collects metrics, use these metrics to determine the efficacy of these models and then enable retraining of the models on the basis of these insights and/or new data. To add to it, all this needs a mechanism that can be automated with the right knobs and dials to allow data science teams to be able to keep a tab and not allow the pipeline to go rogue, which could result in considerable business losses, as these data science models are often linked directly to customer actions.

This problem is very similar to what application development teams face with respect to managing apps and releasing new versions of it at regular intervals with improved features and capabilities. The app dev teams address these with DevOps, which is the industry standard for managing operations for an app dev cycle. To be able to replicate the same to machine learning cycles is not the easiest task.

  Visual: DevOps Process

 

This is where the Azure Machine Learning shines the most. It presents the most complete and intuitive model lifecycle management experience alongside integrating with Azure DevOps and GitHub.

The first task in the ML lifecycle management, after a data scientist has created and validated a model or an ML pipeline, is that it needs to be packaged, so that it can execute where it needs to be deployed. This means that the ML platform needs to enable containerizing the model with all its dependencies, as containers are the default execution unit across scalable cloud services and the IoT edge. Azure Machine Learning provides an easy way for data scientists to be able to package their models with simple commands that can track all dependencies like conda environments, python versioned libraries, and other libraries that the model references so that the model can execute seamlessly within the deployed environment.

The next step is to be able to version control these models. Now, the code generated, like the Python notebooks or scripts can be easily versioned controlled in GitHub, and this is the recommended approach, but in addition to the notebooks and scripts you also need a way to version control the models, which are different entities than the python files. This is important as data scientists may create multiple versions of the model, and very easily lose track of these in search of better accuracy or performance. Azure Machine Learning provides a central model registry, which forms the foundation of the lifecycle management process. This repository enables version control of models, it stores model metrics, it allows for one-click deployment, and even tracks all deployments of the models so that you can restrict usage, in case the model becomes stale or its efficacy is no longer acceptable. Having this model registry is key as it also helps trigger other activities in the lifecycle when new changes appear, or metrics cross a threshold.

Visual: Model Registry in Azure Machine Learning

Once a model is packaged and registered, it’s time to test the packaged model. Since the package is a container, it is most ideal to test it in Azure Container Instances, which provides an easy, cost-effective mechanism to deploy containers. The important thing here is you don’t have to go outside Azure Machine Learning, as it has built strong links to Azure Container Instances within its workspace. You can easily set up an Azure Container Instance from within the workspace or from your IDE, where you’re already using Azure Machine Learning, via the SDK. Once you deploy this container to Azure Container Instances, you can easily inference against this model for testing purposes.

Following a thorough round of testing of the model, it is now time to be able to deploy the model into production. Production environments are synonymous with scale, flexibility and tight monitoring capabilities. This is where Azure Kubernetes Services (AKS) can be very useful for container deployments. It provides scale-out capabilities as it’s a cluster and can be sized to cater to the business’ needs. Again, very much like Azure Container Instances, Azure Machine Learning also provides the capability to set up an AKS cluster from within its workspace or the IDE of choice for the user.

If your models are sufficiently small and don’t need scale-out requirements, you can also take your models to production on Azure Container Instances. Usually, that’s not the case, as models are accessed by end-user applications or many different systems, and such planning for scale always helps. Both Azure Container Instances and AKS provide extensive monitoring and logging capabilities.

Once your model is deployed, you want to be able to collect metrics on the model. You want to ascertain that the model is drifting from its objective and that the inference is useful for the business. This means you capture a lot of metrics and analyze them. Azure Machine Learning enables this tracking of metrics for the model is a very efficient manner. The central model registry becomes the one place where all this hosted.

As you collect more metrics and additional data becomes available for training, there may be a need to be able to retrain the model in the hope of improving its accuracy and/or performance. Also, since this is a continuous process of integrations and deployment (CI/CD), there’s a need for this process to be automated. This process of retraining and effective CI/CD of ML models is the biggest strength of Azure Machine Learning.

Azure Machine Learning integrated with Azure DevOps for you to be able to create MLOps pipelines inside the DevOps environment. Azure DevOps has an extension for Azure Machine Learning, which enables it to listen to Azure Machine Learning’s model registry in addition to the code repository maintained in GitHub for the python notebooks and scripts. This enables to trigger Azure Pipelines based on new code commits into the code repository or new models published into the model repository. This is extremely powerful, as data science teams can configure stages for build and release pipelines within Azure DevOps for the machine learning models and completely automate the process.

What’s more, since Azure DevOps is also the environment to manage app lifecycles it now enables data science teams and app dev teams to collaborate seamlessly and trigger new version of the apps whenever certain conditions are met for the MLOps cycle, as they are the ones often leveraging the new versions of the ML models, infusing them into apps or updating inference call URLs, when desired.

This may sound simple and the most logical way of doing it, but nobody has been able to bring MLOps to life with such close-knit integration into the whole process. Azure Machine Learning does an amazing job of it enabling data science teams to become immensely productive.

Please see the diagrammatic representation below for MLOps with Azure Machine Learning.

Visual: MLOps with Azure Machine Learning

To learn more about MLOps please visit the Azure Machine Learning documentation on MLOps.

Get started now!

This has been a long post so thank you for your patience, but this is just the beginning. As we observe, Azure Machine Learning presents capabilities that make the entire ML lifecycle a seamless process. With these two features, we’re just scratching the surface of its capabilities as there are many more features to help data scientists and machine learning engineers create, manage, and deploy their models in a much more robust and thoughtful manner.

Model interpretability – Understand the model and its behavior.

ONNX runtime support – Deploy models created in the open ONNX format.

Model telemetry collection – Collect telemetry from live running models.

Field programmable gated-array (FPGA) inferencing – Score or featurize image data using pre-trained deep neural networks with blazing fast speed and low cost.

IoT Edge deployment – Deploy model to IoT devices.

And many more to come. Please visit the Getting started guide to start the exciting journey with us!
Quelle: Azure

Customize networking for DR drills: Azure Site Recovery

One of the most important features of a disaster recovery tool is failover readiness. Administrators ensure this by watching out for health signals from the product. Some also choose to set up their own monitoring solutions to track readiness. End to end testing is conducted using disaster recovery (DR) drills every three to six months. Azure Site Recovery offers this capability for replicated items and customers rely heavily on test failovers or planned failovers to ensure that the applications work as expected. With Azure Site Recovery, customers are encouraged to use non-production network for test failover so that IP addresses and networking components are available in the target production network in case of an actual disaster. Even with non-production network, the drill should be the exact replica of the actual failover.

Until now, it has been close to being the replica. The networking configurations for test failover did not entirely match the failover settings. Choice of subnet, network security group, internal load balancer, and public IP address per network interfacing controller (NIC) could not be made. This means that customer had to ensure a particular alphabetical naming convention of subnets in test failover network to ensure the replicated items are failed over as intended. This requirement conflicted with the organizations that enforce naming conventions for Azure resources. Also in case you wished to attach networking components, it was only possible manually post test failover operation. Further, if a customer tests the failover of an entire application via recovery plan, the Azure virtual network selection was applied to all the virtual machines irrespective of the application tier.

Test failover settings for networking resources

DR administrators of Azure Site Recovery now have a highly configurable setup for such operational activities. The network settings required for test failover are available for every replicated item. These settings are optional. If you skip it, old behavior will be applied where you can select the Azure virtual network at the time of triggering test failover.

You can go to Compute and Network blade and choose a test failover network. You can further update all the networking settings for each NIC. Only those settings can be updated that were configured on source at the time of enabling replication. The settings only allow you to choose a networking resource which is already created in the target location. Azure Site Recovery does not replicate the changes on networking resources at source. Read the full guidance on networking customization in Azure Site Recovery documentation.

At the time of initiating test failover via replicated item blade, you will no longer see the dropdown to choose Azure virtual network if the settings are pre-configured. If you initiate test failover via recovery plan, you will still see the dropdown to choose Vnet. However, the Vnet will be applied only to those machines that do not have settings pre-configured.

These settings are only available for Azure machines that are protected by Azure Site Recovery. Test failover settings for VMware and physical machines will be available in a couple of milestones.

Azure natively provides you the high availability and reliability for your mission-critical workloads, and you can choose to improve your protection and meet compliance requirements using the disaster recovery provided by Azure Site Recovery. Getting started with Azure Site Recovery is easy, check out pricing information, and sign up for a free Microsoft Azure trial. You can also visit the Azure Site Recovery forum on MSDN for additional information and to engage with other customers.

Related links and additional content

Set up disaster recovery for Azure virtual machines
Customize networking for test failovers
Learn more about disaster recovery drills

Quelle: Azure

Preview: Server-side encryption with customer-managed keys for Azure Managed Disks

Today we’re introducing the preview for server-side encryption (SSE) with customer-managed keys (CMK) for Azure Managed Disks. Azure customers already benefit from server-side encryption with platform managed keys (PMK) for Azure Managed Disks enabled by default. Customers also benefit from Azure disk encryption (ADE) that leverages the BitLocker feature of Windows and the DM-Crypt feature of Linux to encrypt Managed Disks with customer managed keys within the guest virtual machine.

Server-side encryption with customer-managed keys improves on platform managed keys by giving you control of the encryption keys to meet your compliance needs. It improves on Azure disk encryption by enabling you to use any OS types and images for your virtual machines by encrypting data in the storage service. Server-side encryption with customer-managed keys is integrated with Azure Key Vault (AKV) that provides highly available and scalable, secure storage for RSA cryptographic keys backed by hardware security modules (HSMs). You can either import your RSA keys to Azure Key Vault or generate new RSA keys in Azure Key Vault.

Azure Storage handles the encryption and decryption in a fully transparent fashion using envelope encryption. It encrypts data using an Advanced Encryption Standard (AES) 256 based data encryption key which is in turn protected using your keys stored in Azure Key Vault. You have full control of your keys, and Azure Managed Disks uses system-assigned managed identity in your Azure Active Directory for accessing keys in Azure Key Vault. A user with required permissions in Azure Key Vault must first grant permissions before Azure Managed Disks can access the keys. You can prevent Azure Managed Disks from accessing your keys by either disabling your keys or by revoking access controls for your keys. Moreover, you can track the key usage through Azure Key Vault monitoring to ensure that only Azure Managed Disks or other trusted Azure services are accessing your keys.

To enable customer-managed keys for Azure Managed Disks, you must first create an instance of a new resource type called DiskEncryptionSet, which represents a customer-managed key. You must associate your disks, snapshots, and images with a DiskEncryptionSet to encrypt them with customer-managed keys. There is no restriction on the number of resources that can be associated with the same DiskEncryptionSet.

Availability

Server-side encryption customer-managed keys are available for Standard HDD, Standard SSD, and Premium SSD Managed Disks. You can now perform the following operations in the West Central US region via Azure Compute Rest API version 2019-07-01:

Create a virtual machine from an Azure Marketplace image with OS disk encrypted with server-side encryption with customer-managed keys.
Create a custom image encrypted with server-side encryption with customer-managed keys.
Create a virtual machine from a custom image with OS disk encrypted with server-side encryption with customer-managed keys.
Create data disks encrypted with server-side encryption with customer-managed keys.
Create snapshots encrypted with server-side encryption with customer-managed keys.

We’re going to add support for Azure SDKs and other regions soon.

Getting Started

Please email AzureDisks@microsoft.com to get access to the preview.

Review the server-side encryption with customer-managed keys for Managed Disks preview documentation to learn how to do the following:

Create a virtual machine from an Azure marketplace image with disks encrypted with server-side encryption with customer-managed keys
Create a virtual machine from a custom image with disks encrypted with server-side encryption with customer-managed keys
Create an empty managed disk encrypted with server-side encryption with customer-managed keys and attach it to a virtual machine
Create a new custom image encrypted with server-side encryption with customer-managed keys from a virtual machine with disks encrypted with server-side encryption with customer-managed keys.

Quelle: Azure

Rain or shine: Azure Maps Weather Services will bring insights to your enterprise

Weather: the bane of many motorists, transporters, agriculturalists, retailers, or just about anyone who has to deal with it—which is all of us. That said, we can embrace weather and use weather data to our advantage by integrating it into our daily lives.

Azure Maps is proud to share the preview of a new set of Weather Services for Azure customers to integrate into their applications. Azure Maps is also proud to announce a partnership with AccuWeather—the leading weather service provider, recognized and documented as the most accurate source of weather forecasts and warnings in the world. Azure Maps Weather Services adds a new layer of real-time, location-aware information to our portfolio of native Azure geospatial services powering Microsoft enterprise customer applications.

“AccuWeather’s partnership with Microsoft gives all Azure Maps customers the ability to easily use and integrate authentic and highly accurate weather-based location intelligence and routing into their applications. This is a game-changer.”, says Dr. Joel N. Myers, AccuWeather Founder and CEO. “We are delighted with this collaboration with Microsoft as it will open up new opportunities for organizations–large and small–to benefit from our superior weather data based on their unique needs.”

The power of Azure Maps Weather Services

Bringing Weather Services to Azure Maps means customers now have a simple means of integrating highly dynamic, real-time weather data and visualizations into their applications. There are a multitude of scenarios that require global weather information for enterprise applications. For motorists, we can pick up our phones, or ask a smart speaker about the weather. Our cars can determine the best path for us based on traffic, weather, and personal timing considerations.

Transportation companies can now feed weather information into dynamic routing algorithms to determine the best route conditions for their respective loads. Agriculturalists can have their smart sprinkler systems running connected edge computing informed of incoming rain, saving crops from overwatering, and conserving the delicate resource that is water. Retailers can use predicted weather information to determine the need for high-volume goods, optimizing supply chain.

Did you know that most electrical vehicle batteries lose a percentage of their charge when the temperature dips below freezing? With Azure Maps Weather Services, you can use current or forecasted temperatures to determine your vehicle’s range. Range can determine how far a car can drive along a route, set better expectatoins for estimated arrival times, determine if charging stations are close by, or find hotels that are reachable based on this reduction in battery life. Freezing temperatures also increase the duration a battery will take to charge—meaning more time spent at the charging station.

Having the insight of temperature drops at charging stations means having the ability to calculate the length of time a driver will spend at a charging station; which, in turn, allows charging station owners to recalculate productivity metrics for their respective stations based on weather conditions.

Azure Maps Weather Services in preview

Azure Maps Weather Services are available as a preview with the following capabilities:

Weather Tile API: Fetches radar and infrared raster weather tiles formatted to be integrated into the Azure Maps SDKs. By default, Azure Maps uses vector map tiles for its web SDK (see Zoom Levels and Tile Grid). Use of the Azure Maps SDK is not required and developers are free to integrate the Azure Maps Weather Services into their own Azure Maps applications as needed.
Current Conditions: Returns detailed current weather conditions such as precipitation, temperature, and wind for a given coordinate location. By default, the most current weather conditions will be returned. Observations from the past 6 or 24 hours for a particular location can be retrieved.
Minute Forecast: Request minute-by-minute forecasts for a given location for the next 120 minutes. Users can request weather forecast in the interval of 1, 5 and 15 minutes. The response will include details such as the type of precipitation (including rain, snow, or a mixture of both), start time, and precipitation intensity value.
Hourly Forecast: Request detailed weather forecast by hour for the next 1, 12, 24 (1 day), 72 (3 days), 120 (5 days), and 240 hours (10 days) for the given the given coordinate location. The API returns details such as temperature, humidity, wind, precipitation, and ultraviolet (UV) index.
Quarter-Day Forecast: Request detailed weather forecast by quarter-day for the next 1, 5, 10, or 15 days for a given location. Response data is presented by quarters of the day—morning, afternoon, evening, and overnight. Details such as temperature, humidity, wind, precipitation, and UV index are returned.
Daily Forecast: Returns detailed weather forecast such as temperature, humidity, wind by day for the next 1, 5, 10, 15, 25, or 45 days for a given coordinate location. The API returns details such as temperature, humidity, wind, precipitation, and UV index.
Weather Along Route: Weather along a route API returns hyperlocal (one kilometer or less), up-to-the-minute weather nowcasts, weather hazard assessments, and notifications along a route described as a sequence of waypoints. This includes a list of weather hazards affecting the waypoint or route, and the aggregated hazard index for each waypoint might be used to paint each portion of a route according to how safe it is for the driver. Data is updated every five minutes. The service supplements Azure Maps Route Service that allows you to first request a route between an origin and a destination and use that as an input for Weather Along Route endpoint.

Using the Azure Maps Weather Service along a calculated route (using Azure Maps Route Service), customers can generate weather notifications for waypoints that experience an increase in intensity of a weather hazard. If the vehicle is expected to begin experiencing heavy rain as it reaches a waypoint, a weather notificationwill be generated, allowing the end product to display a heavy rain notification before the driver reaches that waypoint. The trigger for when to display the notification for a waypoint is left up to the product developer and could be based, for example, on a fixed geometry (geofence), or selectable distance to the waypoint.

Azure Maps services are designed to be used in combination with one another to build rich, geospatial applications and insights as part of your Azure Maps account. Azure Maps Weather Service is a new pillar of intelligence added to Azure Maps Location Based Services, Azure Maps Mobility Services, and Azure Maps Spatial Operations, all actuated via the Azure Maps Web, Android SDKs and REST endpoints.

These new weather services are available to all Azure customers, includeing both pay-as-you-go or enterprise agreements. Simply navigate to the Azure Portal, create your Azure Maps account and start using the Azure Maps Weather Service.

We want to hear from you

We are always working to grow and improve the Azure Maps platform and want to hear from you! We’re here to help and want to make sure you get the most out of the Azure Maps platform.

Have a feature request? Add it or vote up the request on our feedback site.
Having an issue getting your code to work? Have a topic you would like us to cover on the Azure blog? Ask us on the Azure Maps forums.
Looking for code samples or wrote a great one you want to share? Join us on GitHub.
To learn more, read the Azure Maps documentation.

Quelle: Azure

Advancing industrial IoT capabilities in Azure Time Series Insights

Late last year, we announced the preview of some of the foundational capabilities of our industrial IoT analytics platform with a scalable time series storage for trending decades of data, semantic model support to describe domain-specific metadata, and enhanced analytics APIs and UX. We are building on the power of this analytics platform with additional new capabilities that will add richness and flexibility, and open up new scenarios for our enterprise IoT customers. Today, we are announcing the following new capabilities:

Warm and cold analytics support that builds on top of our existing preview and provides retention-based data routing between warm and cold stores. Customers can now perform interactive analytics over warm data as well as gain operational intelligence over decades of historical data stored in a customer-owned Azure Data Lake.
A flexible analytics platform that enables attaching a customer-owned Azure Data Lake to Azure Time Series Insights for data archival, thereby allowing customers to have ownership of their IoT data. Customers can connect to and interop across a variety of advanced analytics scenarios such as predictive maintenance and machine learning using familiar technologies including Apache Spark™, Databricks, Jupyter, etc.
Rich query APIs and user experience to support interpolation, new scalar and aggregate functions, categorical variables, scatter plots, and time shifting of time series signals for in-depth analysis.
Significant scale and performance improvements at all layers of the solution including ingestion, storage, query, and metadata/model to support customers’ IoT solution needs.
Azure Time Series Insights Power BI connector that enables customers to take the queries they do in Azure Time Series Insights directly into Power BI to get a unified view of their BI and time series analytics in a single pane of glass.

Azure Time Series Insights continues to provide a scalable pay-as-you-go pricing model enabling customers to tune their usage to suit their business demands and let Azure Time Series Insights analytics platform worry about scaling the infrastructure to meet their growing needs.

A comprehensive analytics platform for Industrial IoT

We released a preview of our first wave of capabilities last year in December. We have since had great customer adoption and feedback that has led us to the preview refresh today.

Our customers span all major industrial IoT segments including manufacturing, automotive, oil and gas, power and utility, smart buildings, and IoT consulting. These customers are telling us that IoT time series analytics is more than just the potential to achieve operational excellence. IoT time series data together with rich contextualization helps them drive dynamic transformation, enabling their businesses to become more agile and data-driven than ever before.

To help maximize the value of time series data and drive this digital revolution, we're updating the Azure Time Series Insights offering to support comprehensive and rich analytics over multi-layered storage, open file format and flexibility to connect to other data services for connected data scenarios, enterprise grade scale and performance, enhanced user experience and SDK support, and out-of-box connectors to data services such as Power BI to enable end-to-end analytics scenarios.

Details of the new features in preview refresh

Comprehensive and rich analytics over multi-layered storage

The majority of industrial IoT customers work with IoT data for a variety of data access scenarios. To satisfy these requirements, Azure Time Series Insights provides scalable multi-layered time series storage for warm and cold data analytics. When a customer provisions Azure Time Series Insights, upon selecting the PAYG pricing option, they can configure Azure Storage as the cold store, as well as enable warm store. Additionally, a customer can choose the retention period (configurable at any time) for the warm store.  Azure Time Series Insights will automatically route ingested data based on the configured retention period to warm store, for example if retention was configured as 30d, as data is being streamed, 30d worth of data is stored in warm store. All data is, by default, routed to customer-owned Azure data lake for purposes of archiving and analytics. Queries done over the configured retention period are always served up from the warm store with no additional input from the user. Queries outside of the retention period are always served up from the cold store. This allows customers to do high volume, interactive, asset-based analytics over warm for monitoring, dashboarding, and troubleshooting scenarios. Customers can continue to do asset-based analytics over decades of cold data stored in their Azure Data Lake for operational intelligence including troubleshooting, golden batch analysis, predictive analytics, etc.

Flexible analytics platform for integrating with first and third party data services

A critical and powerful capability that is unleashed with our cold store is data connectivity to other data solutions for end-to-end scenario coverage. As mentioned earlier, the cold store is a customer-owned Azure Data Lake and is the source of truth for all their IoT data and metadata. Data is stored in an open source, Apache Parquet format for efficient data compression, space, query efficiency, and portability.

Azure Time Series Insights will provide out-of-box connectors for popular and familiar data services that our customers use, for example Apache Spark™ or Databricks for machine learning, and predictive analytics. This is a work in progress and will become available to customers shortly.

As part of this preview refresh, we are releasing the Azure Time Series Insights Power BI connector. This feature is available in the Azure Time Series Insights Explorer user experience through the ‘Export’ option, allowing customers to export the time series queries they create in our user experience directly into the Power BI desktop and view their time series charts alongside other BI analytics. This opens the door to a new class of scenarios for industrial IoT enterprises who have invested in Power BI. It provides a single pane of glass over analytics from various data sources including IoT time series, thereby unlocking significant business and operational intelligence.

Enhanced asset-based analytics API and user experience

Since our preview launch in December last year, we have worked with a number of key IoT enterprise customers to prioritize the set of requirements around query and user experience. The result is the following new capabilities we are announcing as part of our preview refresh today:

Interpolation to reconstruct time series signals from existing data
Discrete signal processing with categorical variables
Trigonometric functions
Scatter plots
Time shifting time series signals to understand data patterns
Model API improvements for hierarchy traversal, time series search, auto-complete, paths, and facets
Improved search and navigation efficiency and continuation token to support query at scale
Improved charting capabilities including support for step interpolation, minimum or maximum shadows, etc.​
Updated model authoring and editing experience
Increased query concurrency to support up to 30 concurrent queries

We have a number of new capabilities coming in this space including support for time weighted averages, additional scalar and aggregate functions, dashboards, etc. over the coming months.

Azure Time Series Insights is committed to our customers’ success

We look forward to continuing to deliver on our commitment of simplifying IoT for our customers and empowering them to achieve more with their IoT data and solutions. For more information, please visit the Azure Time Series Insights product page and documentation. Also, try out the quickstart to begin using Azure Time Series Insights today.

Please provide feedback and suggestions on how we can improve the product and documentation by scrolling down to the bottom of each documentation page, where you can find a button for “product feedback” or sign in to your GitHub account and provide feedback. We value your input and would love to hear from you.
Quelle: Azure

Building retail solutions with Azure IoT Central

Azure IoT Central is an IoT app platform for solution builders that simplifies the challenges of building and deploying scalable and affordable enterprise grade IoT applications. Across the retail industry, the use of connected devices to deliver business performance continues to grow in popularity. New solutions are accelerating business model transformation by connecting manufacturers, supply chains, warehouses, and store shelves to owners, operators, and customers in exciting new ways. Today we’ll discuss our stance on IoT in the retail industry, as well as tell you about just a few of our partners building incredible solutions on Azure IoT Central.

Based on the recent IoT Signals report, our survey of over 3,000 enterprise decision makers across the world, the Retail industry has the highest adoption rate of IoT related solutions at 90 percent, which is higher than Manufacturing, Transportation, Healthcare, or Government. Right now, retail and wholesale companies see top use cases for IoT within their supply chains (64 percent) and inventory optimization (59 percent) and of course leaders across all industries have concerns about security. Yet, we know that retailers and companies along the value chain have a long way to go before reaping all the benefits that IoT will provide.

Updates to IoT Central

Today, we announced updates to Azure IoT Central to help solution builders move beyond proof of concept to building business-critical applications they can brand and sell directly or through Microsoft AppSource. IoT Central can help retail solution builders accelerate development, enabling them to get connected, stay connected and transform their business by managing IoT solutions that deliver IoT data and insights to business applications where decisions are made. For more information, please see our IoT Cental Blog.

We are supporting retail specific solution builders with five IoT Central retail app templates for builders to brand, customize, and make their own apps using extensibility via APIs, data connectors to business applications, repeatability and manageability of their investment through multitenancy, and seamless device connectivity. Get started today with any app template for free and starter materials like a sample operator dashboard and simulated devices to show you what’s possible. In early 2020 updated pricing will help with predictability as you sell your solutions directly to customers or through Microsoft AppSource.

When you are ready to get to customizing and extending, take a look at our rich documentation set, which augments the journey with overviews, tutorials, how-to’s, and industry relevant concepts.

Figure 1, Your brand — Your SaaS – Customize and extend one of these 5 retail app templates to make them your own

Innovative retail partners building their SaaS with IoT Central

Established industry leaders across the retail ecosystem are optimizing omnichannel solutions with IoT Central; delivering IoT insights and actions from the beginning of the supply chain through distribution, warehousing, and into the hands of consumers through storefront or delivery. Learn about what QuEST Global, C.H. Robinson, Dynamics 365 Connected Store, and Footmarks Inc. are doing today.

Digital distribution center solution from Lenovo

In July, Lenovo introduced Lenovo Digital Distribution Center (built with IoT Central) and discussed many of the challenges faced by distribution centers globally, including staffing surges during peak times, labor costs, space constraints, and overall productivity.

Figure 2, illustration of the digital distribution workflow

Figure 3, Architecture diagram of Digital Distribution Center by Lenovo

Today we’ll introduce three more solution builders developing solutions across connected logistics, store analytics, and smart inventory management.

Connected logistics solutions from C.H. Robinson and QuEST Global

The challenges facing global logistics and fleet management continue to grow as more retailers move to just-in-time shipping and warehousing. With the holiday shopping (and shipping) season fast approaching, global shipping and freight transportation provider, C.H. Robinson, is putting IoT Central to work during its busiest time of the year. Intel intelligent gateways and IoT tags managed by IoT Central bring new data and insights into industry leading Navisphere Vision. Jordan Kass, President of TMC, a division of CH Robinson responsible for Navisphere told us; “Navisphere Vision provides global shippers supply chain visibility and a predictive analytics platform. To speed up our deployment, increase our capabilities, and evolve for the future, we are partnering and building new device connections with Azure IoT Central to empower one robust agnostic connection that allows for infinite scalability and speed. This enables us to further optimize and deliver better outcomes—such as improved savings, reliability, and visibility—during these high-stakes holiday shipping months.”

    Figure 4, an example of a Navishphere Vision dashboard for IoT device insights tracking temperature and humidity levels in shipping containers.  

Figure 5, Architecture diagram for Nacisphere Vision by C.H. Robinson

Road safety is a global issue affecting billions of people around the world. QuEST Global's fleet management solution Fleet Tracker aims to reduce roadside issues, using CalAMP OBD2 dongles to deliver real-time location, driving pattern, speed, engine health, and geo fencing; simultaneously managing vehicles nearly anywhere in the world. Maxence Cacheux, Head of Strategic Partnerships from Quest Global told us, “We are delighted with the successful deployment of our fleet management solution built on IoT Central, which enhanced its speed, security, and scalability. Now we are planning for the future when our customers around the world have tens of thousands of connected devices, delivering business transforming insights and actions.”

Figure 6, an example of a dashboard from QuEST Global’s Fleet Tracker solution, deliveirng insights from connected vehicles

Figure 7, Architecture diagram for Fleet Tracker by QuEST Global

 

Store analytics from Dynamics 365

Dynamics 365 Connected Store empowers retailers with real-time observational data to improve in-store performance. From customer movement to the status of products and store devices, Dynamics 365 Connected Store will provide a better understanding of your retail space. Built with IoT Central, Dynamics 365 Connected Store empowers store managers and employees to provide optimal shopping experiences through triggered alerts based on real-time data from video cameras and IoT sensors. This new workflow can significantly improve in-store performance by protecting inventory, increasing profitability, and optimizing the shopping experience in real time.

Figure 8, An example of Dynamics 365 Connected Store dashboard enabling retail staff to visualize the flow of traffic throughout their grocery store using optical IoT sensors

Figure 9, Architecture diagram of Dynamics 365 Connected Store

Smart Inventory Management from Footmarks Inc.

Consumer packaged goods (CPG) manufacturers share many of the same challenges, one of them is getting hundreds or sometimes thousands of custom assets like displays to the correct retail locations, and keeping them in the store for the right amount of time.

When displays don’t reach their pre-determined locations, retailers experience significant loss in sales, a key impact for brands during important buying times. Around the country today, we know a significant portion of point-of-purchase (POP) display programs are not compliant, a problem that Footmarks Inc. is looking to solve through their Smart Tracking app built with IoT Cental, an asset tracking application that delivers previously unavailable insights to CPG’s.

CPG’s can now track the location of their POP assets throughout the entire supply chain and into store execution. Gone are the days of mystery shopping and expensive store visits to get details on your assets.  Shawn Englund, Footmarks Inc.’s CEO is enthusiastic for the future saying,“We are excited to be working with some of the world’s largest CPGs to solve the age-old issue of merchandizing compliance. By adding Azure IoT Central we are able to gain even more insights throughout our CPG partner supply chains and provide actionable insights throughout each of their campaigns.”

Figure 10, An example of a Footmarks dashboard showing POP asset tracking along the supply chain.

Figure 11, Architecture diagram of Smart Tracking by Footmarks Inc.

Getting started

We are at the beginning of an incredible revolution that connects strategy, tools, and devices, and empowers retailers to turn insights into actions. Retail companies around the world are using IoT today to reinvent how they connect to customers, empower employees with the right information, deliver an intelligent supply chain, and informing new business models as individuals and organizations continue to connect billions of new devices to business applications. Here is how you can get started.

1. Start building today on IoT Central

2. Browse the growing list of retail applications in AppSource, devices in the Azure IoT Certified Device Catalog, or contact any of the solution builders discussed today:

Lenovo Digital Distribution Center, email Lenovo for information
C.H. Robinson Navisphere Vision
Footmarks Inc. Smart Tracking, email Footmarks for more information
Quest – Global Fleet Tracker
Dynamics 365 Connected Store, Learn more and request a preview

3. Connect with us at Microsoft Ignite, November 4-8

4. Visit us at the world’s largest Retail trade show, NRF in New York City where you can speak with experts and get hands on, January 12-14
Quelle: Azure

Azure IoT Central: Democratizing IoT for all solution builders

For the last five years, our industry has buzzed with the promises of IoT. IoT has evolved from being a next-horizon term, to a common vernacular employed across industry conversations. In fact, earlier this year we surveyed 3,000 enterprise decision makers across the world and learned that 85% have developed at least one IoT project. Across four major industries (manufacturing, retail, healthcare, and government), more than 84% of enterprise decision makers consider IoT “critical” to success (read the full report here).

Despite this near consensus, the average maturity of production-level IoT projects remains extremely low. Over 90% of companies experience some failure in the proof of concept stage due to concerns and knowledge gaps around how to scale their solutions securely, reliably, and affordably. This finding is not surprising. Scaling a project not only increases the cost, it also introduces significant technical complexity—from knowing how to adapt an architecture as the number of connected devices grows to millions, to ensuring your security remains robust as your breachable footprint expands.

Having worked with thousands of IoT customers, our engineers have encountered these issues time and time again. We used these learnings to evolve Azure IoT Central and help solution builders avoid common pitfalls that prevent many projects from moving beyond the proof of concept stage. We explained these findings in our new report, “8 attributes of successful IoT solutions,” to help IoT solution builders ask the right questions upfront as they design their systems, and help them select the right technology platforms.

A fully managed IoT app platform

Azure IoT Central is our IoT app platform for solution builders (such as ISVs, SIs, and OEMs), to design, deploy, and manage enterprise-grade solutions that they can brand and sell to their customers, either directly or through Microsoft AppSource.

Azure IoT Central provides a complete and robust platform that handles the “plumbing” of IoT solutions. It is by no means an end-to-end solution. The value of IoT Central is brought to life when solution builders leverage it to connect and manage their devices, as well as to extend device insights into their line of business applications. This allows solution builders to spend their time and energy in their area of expertise, transforming their businesses through value-adding and brand-differentiating elements. With whitelabeling, solution builders can go to market with a resulting solution that reflects their brand. While many customers choose to design and build cloud solutions using individual Azure services (a Platform as a Service, or PaaS, approach), Azure IoT Central reduces the cost of building and maintaining a PaaS-based solution by providing a fully managed platform.

Today we’re announcing several major updates to Azure IoT Central. We’re confident these updates will inspire builders to develop industry-leading solutions with the peace of mind that their applications rest on a secure, reliable, and scalable infrastructure–enabling them to connect and manage devices, generate insights, and bring those new insights into their existing applications.

 

New app templates for industry-focused enablement

Today we are releasing 11 new industry app templates, designed to illustrate the types of solutions our partners and customers can build across retail, healthcare, government, and energy.  

Innovative partners using Azure IoT across industries

From startups to established leaders, we are seeing solution builders across industries leverage Azure IoT Central to transform their industries.

One area where we're seeing solution builders use Azure IoT Central to design innovative solutions is healthcare. Every 20 seconds a limb is lost to diabetes. To tackle this issue, Seattle-based startup, Sensoria Health joined forces with leading orthopedic medical footwear manufacturer, Optima Molliter, to launch an IoT solution that enables continuous, remote monitoring of patients recovering from diabetic foot ulcers. Patients and physicians can leverage the Sensoria Core solution, a hub utilizing IoT and artificial intelligence (AI) based on telemetry from Optima footwear, to monitor real-time patient adherence to clinician recommendations via a mobile app.

Physicians can leverage the clinician dashboard, which provides a holistic view of their patient population, to manage patient interactions, understand patient adherence to recommendations over time, and to decide which patients are in most need of care at a given moment. By enabling real-time alerts, physicians can manage care escalation decisions to expedite the healing of foot wounds and reduce the risk of amputations. Azure IoT Central provided the IoT application infrastructure that allowed Sensoria to quickly build a globally available, secure, and scalable IoT solution. Furthermore, Azure IoT Central leverages Azure API for FHIR, enabling Sensoria Health to ensure healthcare interoperability and compliance standards are met when managing the health data provided by EMR systems and from Sensoria Core embedded microelectronic devices. Read the press release to learn more.

We're also seeing well-established solution builders like C.H. Robinson, an American Fortune 500 provider of multimodal transportation services and third-party logistics, are taking advantage of IoT Central. Using Intel intelligent gateways and IoT tags managed by Azure IoT Central, C.H. Robinson has quickly integrated IoT data and insights into its industry-leading Navisphere Vision product. The Navisphere solution is being used by leading retailers including Microsoft’s own supply chain teams to optimize logistics and costs as we prepare to deliver Surface and Xbox products ahead of the holiday season. Jordan Kass, President of TMC, a division of C.H. Robinson responsible for Navisphere described the challenges facing the industry; “Today, Retailers of all sizes need to know where their products are and where they are going … . Building with IoT Central offered us speed, scale, and simplicity to connect devices like Intel’s gateways and IoT tags.”

Vattenfall, a Swedish energy company investing deeply in renewable energy, and Microsoft are collaborating on solutions using Azure IoT Central to address challenges in energy markets to match supply and demand for renewable energy. “The IoT Central app platform has expedited our product development, providing fast and seamless device connectivity and device management, and built-in data storage, rule creation, and dashboarding for our operators,” says, Sebastian Waldenström, Head of Vattenfall’s IoT and Energy Management.

While many of our partners have established industry expertise within verticals, we’ve also seen IoT Central be used by solution builders with horizontal reach, such as Mesh Systems. Mesh Systems is a global expert in asset-tracking solutions, with customer applications spanning retail, logistics, banking, pest control, construction, and much more. “IoT Central helps us do what we do best–only now what used to take 3 months to build now takes 3 days” said Doyle Baxter, strategic alliance manager at Mesh Systems.

These partners and others are on a journey building with IoT Central. Read more about building retail solutions with IoT Central here, and follow along in the coming months as we feature more partner success across other industry segments.

New capabilities for production-level solutions

Expanding IoT Central portfolio with IoT Edge: Businesses can now run cloud intelligence directly on IoT devices at the edge managed by Azure IoT Central. This new feature helps businesses connect and manage Azure IoT Edge devices, deploy edge software modules, publish insights, and take actions at-scale–all from within Azure IoT Central.

Seamless device connectivity with IoT Plug and Play: Solution builders building with Azure IoT Central can select from a range of Azure IoT Pre-Certified Plug and Play devices and quickly connect them to the cloud. Customers can now build production grade IoT solutions within days without having to write a single line of device code, drastically cutting down the time to market and costs.

Range of actions within the platform: Azure IoT Central exposes various levels of extensibility from within the platform. A user can define rules on device data streams that trigger no-code (Microsoft Flow) or low-code actions (Azure Logic Apps). A solution builder could also configure more complex actions, exchanging data with an external service via a Webhook or Azure Functions based action.

Extensibility through data export: Continuous Data Export from Azure IoT Central can be used to integrate data streams directly into Azure PaaS services like Azure Blob Storage for data retention, or Azure Event Hub and Azure Service Bus for building rich processing pipelines for IoT data and insights into business applications, or into storage for Azure Machine Learning.

Public APIs to access features: Solution builders with extensibility needs beyond device data now have access to Central features through our public APIs. Users can develop robust IoT solutions that leverage IoT Central programmatically as the core for device modelling, provisioning, lifecycle management, operations (updating and commanding), and data querying.

Application repeatability: Today, solution builders can use application templates to export their investments and duplicate them for new customers, saving hours of time on configuration and customization.

Manageability and scale through multitenancy: We know that many solution builders need more than just repeatability; they also need manageability to truly scale their investments to customers. Which is why in the coming months, Azure IoT Central will support multitenancy; solution builders can build once and use a multitenancy interface to on-board, configure, and update many customers and organizations globally across regions, offering both device and data sovereignty without sacrificing manageability.

User access control through custom user roles: Organizational complexity varies across customer solution implementations. Custom user roles allow for clearly defined access control to the data as well as actions and configurations within the system. It gives users control over exactly what they need and nothing more.

Device and data scale: Azure IoT Central scales users' data processing pipelines and provides storage to support millions of devices. Solution builders can achieve device scale by seamlessly connecting devices with IoT Plug and Play integration and authoring IoT Central experiences for Plug and Play devices.

Pricing update: In early 2020, we're unveiling a new pricing tier that will make scaling solutions more affordable and will provide more flexibility for solution builders. IoT Central customers will soon be able to select between multiple pricing plans based on their specific message volume needs for their projects. Check back on our pricing page in the coming weeks for more details.

Azure IoT Central: your IoT application platform

Microsoft is investing $5 billion in Azure IoT over the next four years. Our goal is to simplify the journey in IoT, allowing solution builders to bring solutions to market faster, while staying focused on digital transformation.

Azure IoT Central offers a powerful example of how Microsoft continues to deliver on this commitment. By removing the complexity and overhead of setup, management burden, and operational costs, we can accelerate the creation of innovative solutions across all industries. Azure IoT Central provides organizations with the IoT application platform they need to create the next wave of innovation in IoT. And that means a more intelligent and connected world that empowers people and organizations to achieve more. To learn more, visit our product page.
Quelle: Azure