Kategorie: Microsoft Azure

Moving your Azure Virtual Machines has never been easier!

To meet customer demand, Azure is continuously expanding. We’ve been adding new Azure regions and introducing new capabilities. As a result, customers can now move their existing virtual machines (VMs) to new regions while adopting the latest capabilities. There are other factors that prompt our customers to relocate their VMs. For example, you may want to do that to increase SLAs.

In this blog, we will walk you through the steps you need to follow to move your VMs across regions or within the same region.

Why do customers want to move their Azure IaaS Virtual Machines?

Some of the most common reasons that prompt our customers to move their virtual machines include:

•    Geographical proximity: “I deployed my VM in region A and now region B, which is closer to my end users, has become available.”

•    Mergers and acquisitions: “My organization was acquired, and the new management team wants to consolidate resources and subscriptions into one region.”

•    Data sovereignty: “My organization is based in the UK with a large local customer base. As a result of Brexit, I need to move my Azure resources from various European regions to the UK in order to comply with local rules and regulations.”

•    SLA requirements: “I deployed my VMs in Region A, and I would like to get a higher level of confidence regarding the availability of my services by moving my VMs into Availability Zones (AZ). Region A doesn’t have an AZ at the moment. I want to move my VMs to Region B, which is still within my latency limits and has Availability Zones.”

If you or your organization are going through any of these scenarios or you have a different reason to move your virtual machines, we’ve got you covered!

Move Azure VMs to a target region

For any of the scenarios outlined above, if you want to move your Azure Virtual Machines to a different region with the same configuration as the source region or increase your availability SLAs by moving your virtual machines into an Availability Zone, you can use Azure Site Recovery (ASR). We recommend taking the following steps to ensure a successful transition:

1.    Verify prerequisites: To move your VMs to a target region, there are a few prerequisites we recommend you gather. This ensures that you’re creating a basic understanding of the Azure Site Recovery replication, the components involved, the support matrix, etc.

2.    Prepare the source VMs: This involves ensuring the network connectivity of your VMs, certificates installed on your VMs, identifying the networking layout of your source and dependent components, etc.

3.    Prepare the target region: You should have the necessary permissions to create resources in the target region including the resources that are not replicated by Site Recovery. For example, permissions for your subscriptions in the target region, available quota in the target region, Site Recovery’s ability to support replication across the source-target regional pair, pre-creation of load balancers, network security groups (NSGs), key vault, etc.

4.    Copy data to the target region: Use Azure Site Recovery replication technology to copy data from the source VM to the target region.

5.    Test the configuration: Once the replication is complete, test the configuration by performing a failover test to a non-production network.

6.    Perform the move: Once you’re satisfied with the testing and you have verified the configuration, you can initiate the actual move to the target region.

7.    Discard the resources in the source region: Clean up the resources in the source region and stop replication of data.

 

Move your Azure VM ‘as is’

If you intend to retain the same source configuration as the target region, you can do so with Azure Site Recovery. Your virtual machine configuration availability SLAs will be the same before and after the move. A single instance VM after the move will come back online as a single instance VM. VMs in an Availability Set after the move will be placed into an Availability Set, and VMs in an Availability Zone will be placed into an Availability Zone within the target region.

To learn more about the steps to move your VMs, refer to the documentation.
 

Move your Azure virtual machines to increase availability

As many of you know, we offer Availability Zones (AZs), a high availability offering that protects your applications and data from datacenter failures. AZs are unique physical locations within an Azure region and are equipped with independent power, cooling, and networking. To ensure resiliency, there’s a minimum of three separate zones in all enabled regions. With AZs, Azure offers 99.99 percent VM uptime SLA.

You can use Azure Site Recovery to move your single instance VM or VMs in an Availability Set into an Availability Zone, thereby achieving 99.99 percent uptime SLA. You can choose to place your single instance VM or VMs in an Availability Set into Availability Zones when you choose to enable the replication for your VM using Azure Site Recovery. Ideally each VM in an Availability Set should be spread across Availability Zones. The SLA for availability will be 99.99 percent once you complete the move operation. To learn more about the steps to move the VMs and improve your availability, refer to our documentation.

Azure natively provides you with the high availability and reliability you need for your mission-critical workloads, and you can choose to increase your SLAs and meet compliance requirements using the disaster recovery features provided by Azure Site Recovery. You can use the same service to increase availability of the virtual machines you have already deployed as described in this blog. Getting started with Azure Site Recovery is easy – simply check out the pricing information, and sign up for a free Azure trial. You can also visit the Azure Site Recovery forum on  the Microsoft Developer Network (MSDN) for additional information and to engage with other customers.
Quelle: Azure

Maximize throughput with repartitioning in Azure Stream Analytics

Customers love Azure Stream Analytics for its ease of analyzing streams of data in movement, with the ability to set up a running pipeline within five minutes. Optimizing throughput has always been a challenge when trying to achieve high performance in a scenario that can't be fully parallelized. This occurs when you don't control the partition key of the input stream, or your source “sprays” input across multiple partitions that later need to be merged. You can now use a new extension of Azure Stream Analytics SQL to specify the number of partitions of a stream when reshuffling the data. This new capability unlocks performance and aids in maximizing throughput in such scenarios.

The new extension of Azure Stream Analytics SQL includes a keyword INTO that allows you to specify the number of partitions for a stream when performing reshuffling using a PARTITION BY statement. This new keyword, and the functionality it provides, is a key feature to achieve high performance throughput for the above scenarios, as well as to better control the data streams after a shuffle. To learn more about what’s new in Azure Stream Analytics, please see, “Eight new features in Azure Stream Analytics.”

What is repartitioning?

Repartitioning, or reshuffling, is required when processing data on a stream that is not sharded according to the natural input scheme, such as the PartitionId in the Event Hubs case. This might happen when you don’t control the routing of the event generators or you need to scale out your flow due to resource constraints. After repartitioning, each shard can be processed independently of others, and progress without additional synchronization between the shards. This allows you to linearly scale out your streaming pipeline.

You can specify the number of partitions the stream should be split into by using a newly introduced keyword INTO after a PARTITION BY statement, with a strictly positive integer that indicates the partition count. Please see below for an example:

SELECT * INTO [output] FROM [input] PARTITION BY DeviceID INTO 10

The query above will read from the input, regardless of it being naturally partitioned, and repartition the stream tenfold according to the DeviceID dimension and flush the data to output. Hashing of the dimension value (DeviceID) is used to determine which partition shall accept which substream. The data will be flushed independently for each partitioned stream, assuming the output supports partitioned writes, and either has 10 partitions, or can handle an arbitrary number of such.

A diagram of the data flow with the repartition in place is below:

Why and how to use repartitioning?

Use repartitioning to optimize the heavy parts of processing. It will process the data independently and simultaneously on disjoint subsets, even when the data is not naturally partitioned properly on input. The partitioning scheme is carried forward as long as the partition key stays the same.

Experiment and observe the resource utilization of your job to determine the exact number of partitions needed. Remember, Streaming Unit (SU) count, which is the unit of scale for Azure Stream Analytics, must be adjusted so the number of physical resources available to the job can fit the partitioned flow. In general, six SUs is a good number to assign to each partition. In case there are insufficient resources assigned to the job, the system will only apply the repartition if it benefits the job.

When joining two streams of data explicitly repartitioned, these streams must have the same partition key and partition count. The outcome is a stream that has the same partition scheme. Please see below for an example:

WITH step1 AS (SELECT * FROM [input1] PARTITION BY DeviceID INTO 10),
step2 AS (SELECT * FROM [input2] PARTITION BY DeviceID INTO 10)

SELECT * INTO [output] FROM step1 PARTITION BY DeviceID UNION step2 PARTITION BY DeviceID

Specifying a mismatching number of partitions or partition key would yield a compilation error when creating the job.

When writing a partitioned stream to an output, it works best if the output scheme matches the stream scheme by key and count, so each substream can be flushed independently of others. Alternatively, the stream must be merged and possibly repartitioned again by a different scheme before flushing. This would add to the general latency of the processing, as well as the resource utilization and should be avoided.

For use cases with SQL output, use explicit repartitioning to match optimal partition count to maximize throughput. Since SQL works best with eight writers, repartitioning the flow to eight before flushing, or somewhere further upstream, may prove beneficial for the job’s performance. For more information, please refer to the documentation, “Azure Stream Analytics output to Azure SQL Database.”

Next steps

Get started with Azure Stream Analytics and have a look at our documentation to understand how to leverage query parallelization in Azure Stream Analytics.

For any question, join the conversation on Stack Overflow.
Quelle: Azure

Benefits of using Azure API Management with microservices

The IT industry is experiencing a shift from monolithic applications to microservices-based architectures. The benefits of this new approach include:

Independent development and freedom to choose technology – Developers can work on different microservices at the same time and choose the best technologies for the problem they are solving.
Independent deployment and release cycle – Microservices can be updated individually on their own schedule.
Granular scaling – Individual microservices can scale independently, reducing the overall cost and increasing reliability.
Simplicity – Smaller services are easier to understand which expedites development, testing, debugging, and launching a product.
Fault isolation – Failure of a microservice does not have to translate into failure of other services.

In this blog post we will explore:

How to design a simplified online store system to realize the above benefits.
Why and how to manage public facing APIs in microservice-based architectures.
How to get started with Azure API Management and microservices.

Example: Online store implemented with microservices

Let’s consider a simplified online store system. A visitor of the website needs to be able to see product’s details, place an order, review a placed order.

Whenever an order is placed, the system needs to process the order details and issue a shipping request. Based on user scenarios and business requirements, the system must have the following properties:

Granular scaling – Viewing product details happens on average at least 1,000 times more often than placing an order.
Simplicity – Independent user actions are clearly defined, and this separation needs to be reflected in the architecture of the system.
Fault isolation – Failure of the shipping functionality cannot affect viewing products or placing an order.

They hint towards implementing the system with three microservices:

Order with public GET and POST API – Responsible for viewing and placing an order.
Product with public GET API – Responsible for viewing details of a product.
Shipping triggered internally by an event – Responsible for processing and shipping an order.

For this purpose we will use Azure Functions, which are easy to implement and manage. Their event-driven nature means that they are executed on, and billed for, an interaction. This becomes useful when the store traffic is unpredictable. The underlying infrastructure scales down to zero in times of no traffic. It can also serve bursts of traffic in a scenario when a marketing campaign becomes viral or load increases during shopping holidays like Black Friday in the United States.

To maintain the scaling granularity, ensure simplicity, and keep release cycles independent, every microservice should be implemented in an individual Function App.

The order and product microservices are external facing functions with an HTTP Trigger. The shipping microservice is triggered indirectly by the order microservice, which creates a message in Azure Service Bus. For example, when you order an item, the website issues a POST Order API call which executes the order function. Next, your order is queued as a message in an Azure Service Bus instance which then triggers the shipping function for its processing.

Top reasons to manage external API communication in microservices-based architectures

The proposed architecture has a fundamental problem, the way communication from outside is handled.

Client applications are coupled to internal microservices. This becomes especially burdensome when you wish to split, merge, or rewrite microservices.
APIs are not surfaced under the same domain or IP address.
Common API rules cannot be easily applied across microservices.
Managing API changes and introducing new versions is difficult.

Although Azure Functions Proxies offer a unified API plane, they fall short in the other scenarios. The limitations should be addressed by fronting Azure Functions with an Azure API Management, now available in a serverless Consumption tier.

API Management abstracts APIs from their implementation and hosts them under the same domain or a static IP address. It allows you to decouple client applications from internal microservices. All your APIs in Azure API Management share a hostname and a static IP address. You may also assign custom domains.

Using API Management secures APIs by aggregating them in Azure API Management, and not exposing your microservices directly. This helps you reduce the surface area for a potential attack. You can authenticate API requests using a subscription key, JWT token, client certificate, or custom headers. Traffic may be filtered down only to trusted IP addresses.

With API Management can also execute rules on APIs. You can define API policies on incoming requests and outgoing responses globally, per API, or per API operation. There are almost 50 policies like authentication methods, throttling, caching, and transformations. Learn more by visiting our documentation, “API Management policies.”

API Management simplifies changing APIs. You can manage your APIs throughout their full lifecycle from design phase, to introducing new versions or revisions. Contrary to revisions, versions are expected to contain breaking changes such as removal of API operations or changes to authentication.

You can also monitor APIs when using API Management. You can see usage metrics in your Azure API Management instance. You may log API calls in Azure Application Insights to create charts, monitor live traffic, and simplify debugging.

API Management makes it easy to publish APIs to external developers. Azure API Management comes with a developer portal which is an automatically generated, fully customizable website where visitors can discover APIs, learn how to use them, try them out interactively, download their OpenAPI specification, and finally sign up to acquire API keys.

How to use API Management with microservices

Azure API Management has recently become available in a new pricing tier. With its billing per execution, the consumption tier is especially suited for microservice-based architectures and event-driven systems. For example, it would be a great choice for our hypothetical online store.

For more advanced systems, other tiers of API Management offer a richer feature set.

Regardless of the selected service tier, you can easily front your Azure Functions with an Azure API Management instance. It takes only a few minutes to get started with Azure API Management.
Quelle: Azure

How to avoid overstocks and understocks with better demand forecasting

Promotional planning and demand forecasting are incredibly complex processes. Take something seemingly straight-forward, like planning the weekly flyer, and there are thousands of questions involving a multitude of teams just to decide what products to promote, and where to position the inventory to maximize sell-through. For example:

What products do I promote?
How do I feature these items in a store? (Planogram: end cap, shelf talkers, signage etc.)
What pricing mechanic do I use? (% off, BOGO, multi-buy, $ off, loyalty offer, basket offer)
How do the products I'm promoting contribute to my overall sales plan?
How do the products I'm promoting interact with each other? (halo and cannibalization)
I have 5,000 stores, how much inventory of each promoted item should I stock at each store?

If the planning is not successful, the repercussions can hurt a business:

Stockouts directly result in lost revenue opportunities, through lost product sales. This could be a result of customers who simply purchase the desired item from another retailer—or a different brand of the item.
Overstock results in costly markdowns and shrinkage (spoilage) that impacts margin. The opportunity cost of holding non-productive inventory in-store also hurts the merchant. And if inventory freshness is a top priority, poor store allocation can impact brand or customer experience.
Since retailers invest margin to promote products, inefficient promotion planning can be a costly exercise. It’s vital to promote items that drive the intended lift.

Solution

Rubikloud’s Price & Promotion Manager allows merchants and supply chain professionals to take a holistic approach to integrated forecasting and replenishment. The product has three core modules detailed below.

The three modules are:

Learn module: Leverages machine learning to understand how internal and external factors impact demand at a store-sku level, as well as a recommendation framework to improve future planning activities.
Activate module: Allows non-technical users to harness the power of machine learning to better forecast demand and seamlessly integrate forecasts into the supply chain process.
Optimize module: Simulates expected outcomes by changing various demand-driving levers such as promo mechanics, store placement, flyer, halo and cannibalization. The module can quickly reload past campaigns to automate forecast and allocation processes.

In addition, AI automates decision-making across the forecasting lifecycle. The retail-centric approach to forecasting applies novel solutions to more accurately forecast demand. For example, to address new SKUs, the solution uses a new mapping approach to address data scarcity and improve forecast accuracy.

The Price and Promotion Manager solution is built on a cloud-native, SaaS data platform designed to handle enterprise data workloads, covering all aspects of the data journey from ingestion, validation, to transformation into a proprietary data model. Users can seamlessly integrate solution outputs into their supply chain processes. The product design recognizes the challenges faced by category managers and enables a more efficient planning process (for example, a quick view to YoY comp promotions).

Benefits

Addresses data sparsity introduced by new product development and infrequently purchased items to better predict demand through new SKU mapping.
Translates stacked promotions and various promotion mechanics to an effective price, to better model impact on-demand.
Uses hierarchical models to improve forecast accuracy.

Azure Services

Rubikloud’s solution uses the following Azure services

HDInsight: allows Rubikloud to work faster and to have full confidence that they are taking advantage of every possible optimization.
Cosmos DB: provides the convenience of an always-on, reliable, and accessible key/value store. Also provides a reliable database service.
Blob Storage: easy to use and integrates well with HDInsight.
Azure Kubernetes Service (AKS): uses the power of Kubernetes orchestration for all Azure VM customers.

Recommended next steps

Explore how Price & Promotion Manager enables AI powered price and promotion optimization for enterprise retail.
Quelle: Azure

PyTorch on Azure: Deep learning in the oil and gas industry

This blog post was co-authored by Jürgen Weichenberger, Chief Data Scientist, Accenture and Mathew Salvaris, Senior Data Scientist, Microsoft

Drilling for oil and gas is one of the most dangerous jobs on Earth. Workers are exposed to the risk of events ranging from small equipment malfunctions to entire off shore rigs catching on fire. Fortunately, the application of deep learning in predictive asset maintenance can help prevent natural and human made catastrophes.

We have more information than ever on our equipment thanks to sensors and IoT devices, but we are still working on ways to process the data so it is valuable for preventing these catastrophic events. That’s where deep learning comes in. Data from multiple sources can be used to train a predictive model that helps oil and gas companies predict imminent disasters, enabling them to follow a proactive approach.

Using the PyTorch deep learning framework on Microsoft Azure, Accenture helped a major oil and gas company implement such a predictive asset maintenance solution. This solution will go a long way in protecting their staff and the environment.

What is predictive asset maintenance?

Predictive asset maintenance is a core element of the digital transformation of chemical plants. It is enabled by an abundance of cost-effective sensors, increased data processing, automation capabilities, and advances in predictive analytics. It involves converting information from both real-time and historical data into simple, accessible, and actionable insights. This is in order to enable the early detection and elimination of defects that would otherwise lead to malfunction. For example, by simply detecting an early defect in a seal that connects the pipes, we can prevent a potential failure that can result in a catastrophic collapse of the whole gas turbine.

Under the hood, predictive asset maintenance combines condition-based monitoring technologies, statistical process control, and equipment performance analysis to enable data from disparate sources across the plant to be visualized clearly and intuitively. This allows operations and equipment to be better monitored, processes to be optimized, better controlled, and energy management to be improved.

It is worth noting that the predictive analytics at the heart of this process do not tell the plant operators what will happen in the future with complete certainty. Instead, they forecast what is likely to happen in the future with an acceptable level of reliability. It can also provide “what-if” scenarios and an assessment of risks and opportunities.

Figure 1 – Asset maintenance maturity matrix (Source: Accenture)

The challenge with oil and gas

Event prediction is one of the key elements in predictive asset maintenance. For most prediction problems there are enough examples of each pattern to create a model to identify them. Unfortunately, in certain industries like oil and gas where everything is geared towards avoiding failure, the sought-after examples of failure patterns are rare. This means that most standard modelling approaches either perform no better than experienced humans or fail to work at all.

Accenture’s solution with PyTorch and Azure

Although there only exists a small number of failure examples, there exists a wealth of times series and inspection data that can be leveraged.

Figure 2 – Approach for Predictive Maintenance (Source : Accenture)

After preparing the data in stage one, a two-phase deep learning solution was built with PyTorch in stage two. First, a recurrent neural network (RNN) was trained in combination with a long short-term memory (LSTM) architecture which is phase one of stage two. The neural network architecture used in the solution was inspired by Koprinkova-Hristova et al 2011 and Aydin and Guldamlasioglu 2017. This RNN timeseries model forecasts important variables, such as the temperature of an important seal. These forecasts are then fed into a classifier algorithm (random forest) to identify the variable is outside of the safe range and if so, the algorithm produces a ranking of potential causes which experts can examine and address. This effectively enables experts to address the root causes of potential disasters before they occur.

The following is a diagram of the system that was used for training and execution of the solution:  

Figure 3 – System Architecture

The architecture above was chosen to ensure the customer requirement of maximum flexibility in modeling, training, and in the execution of complex machine learning workflows are using Microsoft Azure. At the time of implementation, the services that fit these requirements were HDInsights and Data Science Virtual Machines (DSVM). If the project was implemented today, Azure Machine Learning service would have been used for training/inferencing with HDInsights or Azure Databricks for data processing.

PyTorch was used due to the extreme flexibility in designing the computational execution graphs, and not being bound into a static computation execution graph like in other deep learning frameworks. Another important benefit of PyTorch is that standard python control flow can be used and models can be different for every sample. For example, tree-shaped RNNs can be created without much effort. PyTorch also enables the use of Python debugging tools, so programs can be stopped at any point for inspection of variables, gradients, and more. This flexibility was very beneficial during training and tuning cycles.

The optimized PyTorch solution resulted in faster training time by over 20 percent compared to other deep learning frameworks along with 12 percent faster inferencing. These improvements were crucial in the time critical environment that team was working in. Please note, that the version tested was PyTorch 0.3.

Overview of benefits of using PyTorch in this project:

Training time

Reduction in average training time by 22 percent using PyTorch on the outlined Azure architecture.

Debugging/bug fixing

The dynamic computational execution graph in combination with Python standard features reduced the overall development time by 10 percent.

Visualization

The direct integration into Power BI enabled a high end-user acceptance from day one.

Experience using distributed training

The dynamic computational execution graph in combination with flow control allowed us to create a simple distributed training model and gain significant improvements in overall training time.

How did Accenture operationalize the final model?

Scalability and operationalization were key design considerations from day one of the project, as the customer wanted to scale out the prototype to several other assets across the fleet. As a result, all components within the system architecture were chosen with those as criteria. In addition, the customer wanted to have the ability to add more data sources using Azure Data Factory. Azure Machine Learning service and its model management capability were used to operationalize the final model. The following diagram illustrates the deployment workflow used.

Figure 4 – Deployment workflow

The deployment model was also integrated into a Continuous Integration/Continuous Delivery (CI/CD) workflow as depicted below.

Figure 5 – CI/CD workflow

PyTorch on Azure: Better together

The combination of Azure AI offerings with the capabilities of PyTorch proved to be a very efficient way to train and rapidly iterate on the deep learning architectures used for the project. These choices yielded a significant reduction in training time and increased productivity for data scientists.

Azure is committed to bringing enterprise-grade AI advances to developers using any language, any framework, and any development tool. Customers can easily integrate Azure AI offerings into any part of their machine learning lifecycles to productionize their projects at scale, without getting locked into any one tool or platform.
Quelle: Azure

Azure Stack IaaS – part one

This blog post was co-authored by Daniel Savage, Principal Program Manager, Azure Stack and Tiberiu Radu, Senior Program Manager, Azure Stack.

Azure Stack at its core is an Infrastructure-as-a-Service (IaaS) platform

When we discuss Azure Stack with our customers, they see the value in Azure Stack providing cloud-native capabilities to their datacenters. They see the opportunity to modernize their apps and address the unique solutions Azure Stack can deliver, but they often pause as they ponder where to begin. They wonder how to get value from the investments they have in apps currently running on virtual machines (VM). They wonder, “Does Azure Stack help me here? What if I am not quite ready for Platform-as-a-Service?” These questions are difficult, but the answers become more clear when they understand that Azure Stack at its core is an IaaS platform.

Azure Stack allows customers to run their own instance of Azure in their datacenter. Organizations pick Azure Stack as part of their cloud strategy because it helps them handle situations when the public cloud won’t work for them. The three most common reasons use Azure Stack are because of poor network connectivity to the public cloud, regulatory or contractual requirements, or backend systems that cannot be exposed to the Internet.

Azure Stack has created a lot of excitement around new hybrid application patterns, consistent Azure APIs to simplify DevOps practices and processes, the extensive Azure ecosystem available through the Marketplace, and the option to run Azure PaaS Services locally, such as App Services and IoT Hub. Underlying all of these are some exciting IaaS capabilities and we are so exciting to be kicking off a new blog series to show it off. 

Welcome to the Azure Stack IaaS blog series!

To learn more, please see the below resources:

Azure Stack use cases
Azure IaaS overview

IaaS is more than virtual machines

People often think of IaaS as simply virtual machines, but IaaS is more. When you deploy a VM in Azure or Azure Stack, the machine comes with a software defined network including DNS, public IPs, firewall rules (also called network security groups), and many other capabilities. The VM deployment also creates disks for your VMs on software defined storage running in Blob Storage. In the Azure Stack portal image, you can see how this full software defined infrastructure is displayed after you have deployed a VM:

To learn more, please see below for product overviews:

Azure Virtual Machines
Azure Virtual Networks
Azure Managed Disks
Azure Storage

IaaS is the foundation for PaaS Services

Did you know that the Azure PaaS services are powered by IaaS VMs behind the scenes? As a user you don’t see these VMs, but they deliver the capabilities like Event Hubs or Azure Kubernetes Service (AKS). This same Azure IaaS is the foundation of PaaS in Azure Stack. Not only can you use it to deliver your applications, Azure PaaS services will use IaaS VMs to deliver solutions on Azure Stack.

Take Event Hubs, currently in private preview, as an example. An Azure Stack administrator downloads the Event Hubs resource provider from the Marketplace and installs it. Installation creates a new admin subscription and a set of IaaS resources. The administrator sees things like virtual networks, DNS zones, and virtual machine scale sets in the administration portal:

However, when one of your developers deploys their Event Hub in Azure Stack, they don’t see the behind-the-scenes IaaS VMs and resources in their subscription, they just see the Event Hub:

Modernize your apps through operations

Often people think that application modernization involves writing or changing application code, or that modernization means rearchitecting the entire application. In most cases, the journey starts with small steps. When you run your VMs in Azure or Azure Stack, you can modernize your operations.

In addition to the underlying infrastructure, Azure and Azure Stack offers a full set of integrated and intelligent services. These services support the management for your VMs, self-service capabilities, enhance deployment, and enable infrastructure-as-code. With Azure Stack, you empower your teams. 

Over the next couple of blog posts we will go into more detail about these areas. Here is a chart of the cloud capabilities you can utilize to modernize your IaaS VM operations:

What’s next in this blog series

We hope you come back to read future posts in this blog series. Here are some of our planned upcoming topics:

Fundamentals of IaaS
Start with what you already have
Do it yourself
Pay for what you use
It takes a team
If you do it often, automate it
Protect your stuff
Build on the success of others
Journey to PaaS

Quelle: Azure

Azure IoT Edge runtime available for Ubuntu virtual machines

Azure IoT Edge is a fully managed service that allows you to deploy Azure and third-party services—edge modules—to run directly on IoT devices, whether they are cloud-connected or offline. These edge modules are container-based and offer functionality ranging from connectivity to analytics to storage—allowing you to deploy modules entirely from the Azure portal without writing any code. You can browse existing edge modules in the Azure Marketplace.

Today, we’re excited to offer the open-source Azure IoT Edge runtime preinstalled on Ubuntu virtual machines to make it even easier to get started, simulate an edge device, and scale out your automated testing.

Why use virtual machines?

Azure IoT Edge deployments are built to scale so that you can deploy globally to any number of devices and simulate the workload with virtual devices which is an important step to verify if your solution is ready for mass deployment. The easiest way to do this is by creating simulated devices with Azure virtual machines (VMs) running the Azure IoT Edge runtime to scale your testing from the earliest stages of development—even before you have production hardware.

Azure VMs are:
•    Scalable/automatable: deploy as many as you need
•    Persistent: cloud- managed, rather than locally
•    Flexible: any operating system and elastic resources
•    Easy to use: deploy with simple command line instructions or template

Azure IoT Edge on Ubuntu VM

On first boot, the Azure IoT Edge on Ubuntu VM preinstalls the latest version of the Azure IoT Edge runtime, so you will always have the newest features and fixes. It also includes a script to set the connection string and then restarts the runtime, which can be triggered remotely through the Azure VM portal or Azure command line, allowing you to easily configure and connect the IoT Edge device without starting a secure shell (SSH) or remote desktop session. This script will wait to set the connection string until the IoT Edge client is fully installed so that you don’t have to build that into your automation.

The initial offering is based on Ubuntu Server 16.04 LTS, but other operating systems and versions will be added based on user feedback. We’d love to hear your thoughts in the comments.

Getting started

You can deploy the Azure IoT Edge on Ubuntu VM through the Azure Marketplace, Azure Portal, or Azure Command-line. Let me show you how to use the Azure Marketplace and the Portal.

Azure Marketplace

The quickest way to set up a single instance is to use the Azure Marketplace:

1.  Navigate to the Marketplace with our short link or by searching “Azure IoT Edge on Ubuntu” on the Azure Marketplace

2.  Select “GET IT NOW” and then “Continue” on the next dialog.

3.  Once in the Azure Portal, select “Create” and follow the wizard to deploy the VM.

If it’s your first time trying out a VM, it’s easiest to use a password and enable the SSH in the public inbound port menu.
If you have a resource intensive workload, you should upgrade the virtual machine size by adding more CPUs and/or memory.

4.  Once the virtual machine is deployed, configure it to connect to your IoT Hub by:

Copying your device connection string from your IoT Edge device created in your IoT Hub (You can follow the “Register a new Azure IoT Edge device from the Azure portal” how-to guide if you aren’t familiar with this process)
Selecting your newly created virtual machine resource from the Azure Portal and opening the “run command” option

Select the “RunShellScript” option

Execute the script below via the command window with your device connection string:/etc/iotedge/configedge.sh “{device_connection_string}”
Select “Run”
Wait a few moments, and the screen should then provide a success message indicating the connection string was set successfully.

5.  Voila! Your IoT Edge on Ubuntu VM is now connected to IoT Hub

Azure portal

If you’re already working in the Azure portal, you can search for “Azure IoT Edge” and select “Ubuntu Server 16.04 LTS + Azure IoT Edge runtime” to begin the VM creation workflow. From there, complete steps 3-5 in the Marketplace instructions above.

 

If you’d like to learn how you can deploy these virtual machines at scale, check out the “Deploy from Azure CLI” section in the Run Azure IoT Edge on Ubuntu Virtual Machines article.

Now that you have created an IoT Edge device with your virtual machine and connected it to your IoT Hub, you can deploy modules to it like any other IoT Edge device. For example, if you go to the IoT Edge Module Marketplace and select the “Simulated Temperature Sensor,” you can deploy this module to the new device and see data flowing in just a few clicks! Next, try deploying your own workloads to the virtual machine and let us know how we can further simplify your IoT Edge testing experience in the comments section below or on user voice. 

Get started with Azure IoT Edge on Ubuntu virtual machines today!
Quelle: Azure

Controlling costs in Azure Data Explorer using down-sampling and aggregation

Azure Data Explorer (ADX) is an outstanding service for continuous ingestion and storage of high velocity telemetry data from cloud services and IoT devices. Leveraging its first-rate performance for querying billions of records, the telemetry data can be further analyzed for various insights such as monitoring service health, production processes, and usage trends. Depending on data velocity and retention policy, data size can rapidly scale to petabytes of data and increase the costs associated with data storage. A common solution for storage of large datasets for a long period of time is to store the data with differing resolution. The most recent data is stored at maximum resolution, meaning all events are stored in raw format. While the historic data is stored at reduced resolution, being filtered and/or aggregated. This solution is often used for time series databases to control hot storage costs.

In this blog, I’ll use the GitHub events public dataset as the playground. For more information read about how to stream GitHub events into your own ADX cluster by reading the blog, “Exploring GitHub events with Azure Data Explorer.” I’ll describe how ADX users can take advantage of stored functions, the “.set-or-append” command, and the Microsoft Flow Azure Kusto connector. This will help you to create and update tables with filtered, down-sampled, and aggregated data for controlling storage costs. The following are steps which I performed.

Create a function for down-sampling and aggregation

The ADX demo11 cluster contains a database named GitHub. Since 2016, all events from GHArchive have been ingested into the GitHubEvent table and now total more than 1 billion records. Each GitHub event is represented in a single record with event-related information on the repository, author, comments, and more.

Initially, I created the stored function AggregateReposWeeklyActivity which counts the total number of events in every repository for a given week.

.create-or-alter function with (folder = "TimeSeries", docstring = "Aggregate Weekly Repos Activity”)
AggregateReposWeeklyActivity(StartTime:datetime)
{
let PeriodStart = startofweek(StartTime);
let Period = 7d;
GithubEvent
| where CreatedAt between(PeriodStart .. Period)
| summarize EventCount=count() by RepoName = tostring(Repo.name), StartDate=startofweek(CreatedAt)
| extend EndDate=endofweek(StartDate)
| project StartDate, EndDate, RepoName, EventCount
}

I can now use this function to generate a down-sampled dataset of the weekly repository activity. For example, using the AggregateReposWeeklyActivity function for the first week of 2017 results in a dataset of 867,115 records.

Using Kusto query, create a table with historic data

Since the original dataset starts in 2016, I formulated a program that creates a table named ReposWeeklyActivity and backfills it with weekly aggregated data from the GitHubEvent table. The query runs in parallel ingestion of weekly aggregated datasets using the “.set-or-append” command. The first ingestion operation also creates the table that holds the aggregated data.

.show table GithubEvent details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

.show table ReposWeeklyActivity details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

Code sample:
using Kusto.Data.Common;
using Kusto.Data.Net.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace GitHubProcessing
{
class Program
{
static void Main(string[] args)
{
var clusterUrl = "https://demo11.westus.kusto.windows.net:443;Initial Catalog=GitHub;Fed=True";
using (var queryProvider = KustoClientFactory.CreateCslAdminProvider(clusterUrl))
{
Parallel.For(
0,
137,
new ParallelOptions() { MaxDegreeOfParallelism = 8 },
(i) =>
{
var startDate = new DateTime(2016, 01, 03, 0, 0, 0, 0, DateTimeKind.Utc) + TimeSpan.FromDays(7 * i);
var startDateAsCsl = CslDateTimeLiteral.AsCslString(startDate);
var command = $@"
.set-or-append ReposWeeklyActivity <|
AggregateReposWeeklyActivity({startDateAsCsl})";
queryProvider.ExecuteControlCommand(command);

Console.WriteLine($"Finished: start={startDate.ToUniversalTime()}");
});
}
}
}
}

Once the backfill is complete, the ReposWeeklyActivity table will contain 153 million records.

Configure weekly aggregation jobs using Microsoft Flow and Azure Kusto connector

Once the ReposWeeklyActivity table is created and filled with the historic data, we want to make sure it stays updated with new data appended every week. For that purpose, I created a flow in Microsoft Flow that leverages Azure Kusto connector to ingest aggregation data on a weekly basis. The flow is built of two simple steps:

Weekly trigger of Microsoft Flow.
Use of “.set-or-append” to ingest the aggregated data from the past week.

For additional information on using Microsoft Flow with Azure Data Explorer see the Azure Kusto Flow connector.

Start saving

To depict the cost saving potential of down-sampling, I’ve used “.show table <table name> details” command to compare the size of the original GitHubEvent table and the down-sampled table ReposWeeklyActivity.

.show table GithubEvent details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

.show table ReposWeeklyActivity details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

The results, summarized in the table below, show that for the same time frame the down-sampled data is approximately 10 times smaller in record count and approximately 180 times smaller in storage size.

 

Original data

Down-sampled/aggregated data

Time span

2016-01-01 … 2018-09-26

2016-01-01 … 2018-09-26

Record count

1,048,961,967

153,234,107

Total size on disk (indexed and compressed)

725.2 GB

4.38 GB

Converting the cost savings potential to real savings can be performed in various ways. A combination of the different methods are usually most efficient in controlling costs.

Control cluster size and hot storage costs: Set different caching policies for the original data table and down-sampled table. For example, 30 days caching for the original data and two years for the down-sampled table. This configuration allows you to enjoy ADX first-rate performance for interactive exploration of raw data, and analyze activity trends over years. All while controlling cluster size and hot storage costs.
Control cold storage costs: Set different retention policies for the original data table and down-sampled table. For example, 30 days retention for the original data and two years for the down-sampled table. This configuration allows you to explore the raw data and analyze activity trends over years while controlling cold storage costs. On a different note, this configuration is also common for meeting privacy requirements as the raw data might contain user-identifiable information and the aggregated data is usually anonymous.
Use the down-sampled table for analysis: Running queries on the down-sampled table for time series trend analysis will consume less CPU and memory resources. In the example below, I compare the resource consumption of a typical query that calculates the total weekly activity across all repositories. The query statistics shows that analyzing weekly activity trends on the down-sampled dataset is approximately 17 times more efficient in CPU consumption and approximately eight times more efficient in memory consumption.

Running this query on the original GitHubEvent table consumes approximately 56 seconds of total CPU time and 176MB of memory.

The same calculation on the aggregated ReposWeeklyActivity table consumes only about three seconds of total CPU time and 16MB of memory.

Next steps

Azure Data Explorer leverages cloud elasticity to scale out to petabyte-size data, depict exceptional performance, and handle high query workloads. In this blog, I’ve described how to implement down-sampling and aggregation to control the costs associated with large datasets.

To find out more about Azure Data Explorer you can:

Try Azure Data Explorer in preview now.
Find pricing information for Azure Data Explorer.
Access documentation for Azure Data Explorer.

Quelle: Azure

Actuating mobility in the enterprise with new Azure Maps services and SDKs

The mobility space is at the forefront of the most complex challenges faced by cities and urban areas today. The movement of people and things is as much a driver of opportunity as it is an agent of chaos, aggravating existing challenges of traffic, pollution, and unbalanced livelihoods. Today, Azure Maps is continuing to expand the offerings of our platform, introducing a new set of capabilities in the form of SDKs and cloud-based services, to enable enterprises, partners, and cities to build the solutions that will help visualize, analyze, and optimize these mobility challenges.

The services we’re introducing are designed exclusively for the needs of the modern enterprise customer – powerful, real-time analytics and seamless cross-screen experiences, fortified by robust security services.

First, we’re officially moving the following services from public preview to general availability: Route Range {Isochrones}, Get Search Polygon, and Satellite and Hybrid Imagery. Furthermore, we’re introducing multiple new services. We’re enhancing our map canvas by introducing a stunning set of natural earth map tiles and an image compositor to make interaction with our maps more aesthetic, useful, and powerful. We’re also introducing Spatial Operations services that offer powerful analytics used by mobility applications and other industries today, as well as a new Android SDK and a Web SDK that equip Azure customers with the tools necessary to make smarter, faster, more informed decisions. And because privacy and security are top of mind, Azure Maps is now natively integrated with Azure Active Directory, making access to our services more secure while enabling roles and restrictions to our customers.

These services will provide Azure customers with the ability to offload map data processing and hosting costs all while getting the benefits of a rich set of maps and mapping services, securely with the fastest map data refresh rate available today. This refresh rate of data and services is bolstered by the recently announced partnership with TomTom who has committed to moving their map-making workloads to the Azure cloud significantly shortening the time from impetus to end user.

Android SDK (Preview)

While the Azure Maps Web SDK can work within a web control on mobile platforms, many developers prefer native support to interoperate with other native components and have these capabilities in native code… Java != JavaScript. In support of our customers who rely on applications running on Android, Azure Maps is distributing an Android SDK, complete with rendering maps and traffic, drawing, event handling, and using the variety of our map canvases. You can also connect to other Azure Maps services such as Search and Routing though the Azure Maps services APIs.

Spatial Operations (Preview)

Data analysis is central to the Internet of Things (IoT). Azure Maps Spatial Operations will take location information and analyze it on the fly to help inform our customers of ongoing events happening in time and space, enabling near real-time analysis and predictive modeling of events. Today Spatial Operations includes the following services:

Geofencing. A geofence is an “invisible fence” around a particular area. These “fences” exist in coordinates in the shape of customizable polygons and can be associated with temporal constraints so that fences are evaluated only when relevant. Furthermore, you can store a geofence Azure Maps Data Service (more on that below). With Azure Event Grid integration, you can create notifications whenever the position of an object changes with respect to a geofence – including entry, exit, or changing proximity to a geofence.

This has multiple applications. In transportation and logistics, it can be used to create alerts when cargo ships have arrived at waypoints along routes – critical for creating delivery alerts as well as for anticipating incidents of piracy. For drones, a geofence could enforce the limitation of where a drone is permitted to fly.

However, geofencing isn’t limited to transportation. In agriculture, it can enable real-time notifications when a herd has left a field. In construction, you can receive a notification when expensive equipment leaves a construction site that it shouldn’t be leaving or when it is parked in an area that may cause damage to equipment. It can also be used to warn site visitors when entering hazardous zones on sites, as implemented by Scottish IoT technology firm, Tagdat. In services, vending machines in college dorms often disappear. The servicing company can be notified if the machine leaves the premises. Geofencing is an amazingly powerful service to provide notifications and analytics when objects are moving. And, even when objects aren’t moving and should be!

Another example of a customer that is using this service is GovQA who state, “We use geofencing to identify when a requester is trying to submit a request which is outside the customer's pre-defined limits. The limits are defined by the city or county during the set up of the system. This allows the city to correctly handle the request and communicate with the requester appropriately using rules and configurations within GovQA."

Buffer. Build an area around points, lines, and polygons based on a given distance. Define the area in proximity of powerlines that should be kept clear from vegetation or create route buffers in fleet management for managing route deviations.

Closest Point. Returns the closest points between a base point and a given collection of points. It can be used to quickly identify the closest stores, charging stations, or in mobility scenarios, it could be used to identify the closest devices.

Great-circle distance. Returns the great-circle or shortest distance between two points on the surface of a sphere. In the context of drone delivery services, this API can be used to calculate the distance between an origin and destination as the crow flies taking into account the curvature of the earth so that a accurate time estimate for the delivery can be taken into account to optimize operations.

Point in Polygon. Returns a Boolean indicating whether the location is inside a given set of Polygon and MultiPolygon geometries. For example, the point in polygon API can be used to determine whether a home for sale is in the preferred area of customers.

Data Service (Preview)

Data is an imperative for maps, and bringing customer data closer to the Azure Maps service will reduce latency, increase productivity, and create powerful, new scenarios to light up in your applications. As such, Azure Maps will now have the ability for customers to upload and store up to 50MB of geospatial data for use with other Azure Maps services, such as geofencing or image composition. 

Azure Active Directory Integration (Preview)

Security and role-based access control have been of paramount concern for the modern enterprise. As such, we’re proud to announce that Azure Active Directory (AD) is now a core capability of Azure Maps. Use Azure AD to protect your customers’ information and implement secure access by providing role-based access control (RBAC).  Whether you have public applications or applications requiring a login, Azure AD and Azure Maps will support your security needs by authenticating your applications and Azure AD user(s). Additionally, this Azure AD implementation supports managed identities for Azure resources which provide Azure services (Azure App service, Azure Functions, Virtual Machines, etc.) with an automatically managed identity that can be authorized for access to Azure Maps services. 

Azure Maps Web SDK 2.0

Today we’re announcing a new module for accessing Azure Maps services to use in conjunction with the Azure Map Control. The new Service Module allows you to natively work directly with the Azure Maps services. This new module, plus the aforementioned adoption of Azure Active Directory, warranted the need for creating a new version and encapsulating them into a single Web SDK. Henceforth, we’ll containerize our services for web developers into the Azure Maps Web SDK 2.0. Note that the Azure Map Control 1.x will continue to be operational. However, we will innovate on 2.0 moving forward. The upgrade path for 1.x to 2.0 is quite simple by changing the version number! As a part of the new Azure Maps Web SDK 2.0 we’re also including some new client features:

Azure Active Directory (AAD). Azure Maps now natively supports Azure Active Directory to keep your access to Azure Maps secure. With native AAD integration, ensure your access is protected when your applications call Azure Maps.

Services Module. The new Services Module adds support for AAD and a much cleaner API interface for accessing Azure Maps services. It works with both the Web SDK and also in NodeJS. Being part of the Azure family of products, the Azure Maps Services Module was designed to align with an initiative to unify Azure SDKs and was required in order to add support for AAD.

Stroke gradients. There are times in location services development when you’d want to have gradient colors throughout the stroke of a line. Azure Maps Web SDK 2.0 now natively supports the ability to fill a line with a gradient of colors to show transition from one segment of a line to the next. As an example, these gradient lines can represent changes over time and distance, or different temperatures across a connected line of objects.

Shaded Relief map style. Below you’ll read about the new Shaded Relief Map Style. This beautiful, new map style is available immediately in the Azure Maps Web SDK 2.0.

Polygon Fill Patterns. Representations of polygons on a map can be done in a plethora of ways. In many scenarios, there will be a need to create polygons on the map. With the Azure Maps Web SDK 2.0 there is native control for shapes, borders, and fills. Now, we natively support patterns as a fill in addition to a single color fill. Patterns provide a unique way to highlight a specific area to really make it standout, especially if that area is surrounded by other color-shaded polygons. As an example, patterns can be used to show an area in transition, an area significantly different from other areas (such as financially, population, or land use), or areas highlighting facets of mobility such as no-fly zones or areas where permits are required.

Shaded Relief map style

Azure Maps comes complete with a few map styles including the road, dark gray, night, and satellite/hybrid. We’re adding a new map style – Shaded Relief – to complement the existing map styles. Shaded Relief is just that – an elegantly designed map canvas complete with the contours of the Earth. The Azure Maps Web SDK comes complete with the Shaded Relief canvas and all functions work seamlessly atop of it.

Image composition

Azure Maps is introducing a new image compositor that allows customers to render raster map images annotated with points, lines, and polygons. In many circumstances you can submit your request along with your respective point data to render a map image. For more complex implementations, you’ll want to use the map image compositor in conjunction with data stored in the aforementioned Azure Maps Data Service.

We always appreciate feedback from our customers. Feel free to comment below or post questions to Stack Overflow or our Azure Maps Feedback Forums.
Quelle: Azure