Kategorie: Microsoft Azure

Azure Stack IaaS – part seven

If you do it often, automate it

In the virtualization days, before cloud and self-service, it took a while to get all the approvals, credentials, virtual LANs (VLANs), logical unit numbers (LUNs), etc. It took so long, that the actual creation part was easy. When cloud came along with self-service, not only was it easier to create a virtual machine (VM) without relying on others, but it changed our thinking about whether VMs were precious or disposable. At the same time developers were moving to a world of continuous delivery to serve their customers who expect apps with a constant stream of new features. This is the reason all real clouds provide automation APIs to quickly create VMs and other resources, including the infrastructure they rely on. This is often called “Infrastructure as code.” Azure’s API is governed by the Azure Resource Manager (ARM). When you set up Azure Stack, you get your own private instance of ARM.

In this blog post I will cover the automation options in your Cloud IaaS toolkit.

Azure portal

The best place to learn this is from first completing a VM deployment through the portal. Azure Stack provides the same portal as Azure. When you get to the last confirmation page, click the Download Template link. This will show you the template that will be used to deploy the VM you just specified on the previous screens.

As you look through the template – which is in JSON format – you can see how the virtual machine, network, and storage is defined. You’ll see an OS profile and hardware profile for your VM. Using this template, you could deploy this same VM over and over. This is perfect if you’re helping your developers with Continuous Integration and Delivery (CI/CD).

One thing about an ARM template is it is not a procedural script like you might get with standard automation tools. This template creates the resources as a single deployment transaction. ARM will either get to the goal state or the deployment will fail. If it fails, you have a chance to fix failure condition and move forward or delete the deployment and start over. This way you don’t get something other than what you specify.

Once you save your template, you can redeploy it in the portal, using the Template deployment item in the marketplace:

Learn more:
Azure Resource Manager overview
Quickstart: Create templates using the Azure Portal
Deploy resources in the portal using a custom template

Visual Studio and Visual Studio Code

Visual Studio can be scary for an infrastructure person, but if you can master some simple things, you can really help your team down the road of automation. The biggest thing that Visual Studio provides is the real-time feedback that you’re authoring the ARM JSON template correctly in terms of syntax. You don’t get this in Notepad. Get started by creating a new Azure Resource Group project:

A great way to start authoring a template is to use Quickstart templates. Azure Stack has a number of Quickstart templates you can use. When you start your new project in Select Azure Template, pick Azure Stack Quickstart from the drop-down list.

Visual Studio not only helps you author the template, it allows you to deploy the template directly to Azure Stack. When you sign into Visual Studio, all of your subscriptions in both Azure and Azure Stack show up as deployment targets. Since I use a number of Azure Stack environments, I have lots of deployment options:

Another way to create these templates is in Visual Studio Code. The ARM template is a JSON file, so you need a good lightweight authoring tool to work on the JSON file. Visual Studio Code (VS Code) is a great option.

After installing VS Code, you need to add the Azure Resource Manager Tools extension. This extension adds many features that simplify template authoring that help you manage variables, parameters, and resource blocks with features like including formatting and color coding. Check it out in the image below:

Learn more:
Install Visual Studio and Connect to Azure Stack
Deploy templates to Azure Stack using Visual Studio
Create a template in Visual Studio
Create a template in Visual Studio Code

PowerShell and Azure command-line interface

The Portal and Visual Studio are both deployment options for your template. PowerShell and the Azure command-line interface (CLI) are two other options. Since Azure Stack is your own private instance of the Azure Resource Manager, you need to connect to your unique instance.

To connect to Azure Stack with PowerShell, use the Add-AzureRMEnvironment cmdlet, specifying the ARM endpoint for your environment. Depending on how your Azure Stack environment has been set up, you will either authenticate using your Azure Active Directory credentials or your organization’s Azure Directory (referred to in the documentation as ADFS).

Azure CLI is Microsoft's cross-platform command-line experience for managing Azure resources. It works on Mac, Linux, and Windows. In Azure you can run the CLI in Cloud Shell. You can use Azure CLI to connect to Azure Stack and deploy IaaS templates. Just like PowerShell, you need to first connect to your Azure Stack’s unique ARM endpoint. For CLI you use the az cloud register command. To deploy your ARM template you use the az group deployment create command.

Please note: We have not implemented Cloud Shell on Azure Stack yet. Cloud Shell allows you to run the Azure CLI directly inside your browser. If you would like to see this, make sure you put in a vote on Azure Cloud Shell UserVoice.

Learn more:
Install PowerShell for Azure Stack
Connect to Azure Stack with PowerShell
Deploy in Azure Stack with PowerShell
Use Azure CLI on Azure Stack
Deploy Azure Resource Manager Templates with Azure CLI

The cloud is for automation

The more people use clouds like Azure and Azure Stack, the less they use the portal and the more they use automation. There is an automation option in Azure and Azure Stack for all your needs. Say goodbye to virtualization and say hello to Cloud IaaS with your automation toolkit.

In this blog series

We hope you come back to read future posts in this blog series. Here are some of our past and upcoming topics:

Azure Stack at its core is an Infrastructure-as-a-Service (IaaS) platform
Start with what you already have
Fundamentals of IaaS
Protect your stuff
Do it yourself
Pay for what you use
Build on the success of others
Journey to PaaS

Quelle: Azure

Deploy a FHIR sandbox in Azure

This blog post was authored by Michael Hansen, Senior Program Manager, Microsoft Azure.

In connection with HIMSS 2019, we announced the Azure API for FHIR, which provides our customers with an enterprise grade, managed FHIR® API in Azure. Since then, we have been busy improving the service with new configuration options and features. Some of the features we have been working on include authentication configuration and the SMART on FHIR Azure Active Directory Proxy, which enable the so-called SMART on FHIR EHR launch with the Azure API for FHIR.

We have developed a sandbox environment that illustrates how the service and the configuration options are used. In this blog post, we focus on how to deploy the sandbox in Azure. Later blog posts will dive into some of the technical details of the various configuration options.

The Azure API for FHIR team maintains a GitHub repository with sample applications. It is maintained by the product engineering team to ensure that it works with the latest features of the Azure API for FHIR. The repository contains a patient dashboard application, Azure Function that will load patient data generated with Synthea, and example templates for SMART on FHIR applications:

Deployment instructions

The repository contains fully automated PowerShell scripts that you can use to deploy the sandbox scenario. The deployment script will create Azure Active Directory application registrations and a test user. If you do not want to create these Azure Active Directory objects in the tenant associated with your Azure subscription, we recommend you create a separate Azure Active Directory tenant to use for data plane access control.

The deployment script is written for PowerShell and uses the AzureAd PowerShell module. If you don’t have access to PowerShell on your computer, you can use the Azure Cloud Shell. In the cloud shell, you can deploy the sandbox environment with:

# Clone source code repository
cd $HOME
git clone https://github.com/Microsoft/fhir-server-samples
cd fhir-server-samples/deploy/scripts

# Log in to Azure AD:
Connect-AzureAd -TenantDomain <mytenantdomain>.onmicrosoft.com

# Connect to Azure Subscription
Login-AzureRmAccount

# Selection subscription
Select-AzureRmSubscription -SubsciptionName “Name of your subscription”

# Deploy Sandbox
.Create-FhirServerSamplesEnvironment.ps1 -EnvironmentName <NameOfEnvironment> -EnvironmentLocation westus2 -AdminPassword $(ConvertTo-SecureString -AsPlainText -Force "MySuperSecretPassword")

It will take around 5 minutes to deploy the environment. The deployment script will create a resource group with the same name as the environment. In there, you will find all the resources associated with the sandbox.

Loading synthetic data

The environment resource group will contain a storage account with a container named “FhirImport.” If you upload Synthea patient bundles to this storage account, they will be ingested.

Using the patient dashboard

There are two versions of the patient dashboard, they can be located at:

https://<NameOfEnvironment>dash.azurewebsites.net: This is an ASP.NET patient dashboard. The GitHub repository contains the source code for this patient dashboard.
https://<NameOfEnvironment>js.azurewebsites.net: This is a single page JavaScript application. The source code is also in the GitHub repository.

When you navigate to either of those URLs, you will be prompted to log in. The administrator user is created by the deployment script and will have the username <NameOfEnvironment>-admin@<mytenantdomain>.onmicrosoft.com and the password is whatever you chose it to be during deployment. If you have uploaded some patients using the Synthea uploader, you should be able to display a list of patients. This shows the view in the JavaScript dashboard.

You can click details on a specific patient to get more information:

You can also use the links for the SMART on FHIR applications to get the growth chart application this patient:

The sandbox provides other useful tools. As an example, the “About me” link will provide you with details about the FHIR endpoint including a token that can be used to access the FIR API using tools like Postman.

Deleting the sandbox

When you are done exploring the Azure API for FHIR and the FHIR sandbox, it is easily deleted with:

.Delete-FhirServerSamplesEnvironment.ps1 -EnvironmentName <NameOfEnvironment>

FHIR® is the registered trademark of HL7 and is used with the permission of HL7
Quelle: Azure

Building recommender systems with Azure Machine Learning service

Recommendation systems are used in a variety of industries, from retail to news and media. If you’ve ever used a streaming service or ecommerce site that has surfaced recommendations for you based on what you’ve previously watched or purchased, you’ve interacted with a recommendation system. With the availability of large amounts of data, many businesses are turning to recommendation systems as a critical revenue driver. However, finding the right recommender algorithms can be very time consuming for data scientists. This is why Microsoft has provided a GitHub repository with Python best practice examples to facilitate the building and evaluation of recommendation systems using Azure Machine Learning services.

What is a recommendation system?

There are two main types of recommendation systems: collaborative filtering and content-based filtering. Collaborative filtering (commonly used in e-commerce scenarios), identifies interactions between users and the items they rate in order to recommend new items they have not seen before. Content-based filtering (commonly used by streaming services) identifies features about users’ profiles or item descriptions to make recommendations for new content. These approaches can also be combined for a hybrid approach.

Recommender systems keep customers on a businesses’ site longer, they interact with more products/content, and it suggests products or content a customer is likely to purchase or engage with as a store sales associate might. Below, we’ll show you what this repository is, and how it eases pain points for data scientists building and implementing recommender systems.

Easing the process for data scientists

The recommender algorithm GitHub repository provides examples and best practices for building recommendation systems, provided as Jupyter notebooks. The examples detail our learnings on five key tasks:

Data preparation – Preparing and loading data for each recommender algorithm
Modeling – Building models using various classical and deep learning recommender algorithms such as Alternating Least Squares (ALS) or eXtreme Deep Factorization Machines (xDeepFM)
Evaluating – Evaluating algorithms with offline metrics
Model selection and optimization – Tuning and optimizing hyperparameters for recommender models
Operationalizing – Operationalizing models in a production environment on Azure

Several utilities are provided in reco utils to support common tasks such as loading datasets in the format expected by different algorithms, evaluating model outputs, and splitting training/test data. Implementations of several state-of-the-art algorithms are provided for self-study and customization in an organization or data scientists’ own applications.
In the image below, you’ll find a list of recommender algorithms available in the repository. We’re always adding more recommender algorithms, so go to the GitHub repository to see the most up-to-date list.

 

Let’s take a closer look at how the recommender repository addresses data scientists’ pain points.

It’s time consuming to evaluate different options for recommender algorithms

One of the key benefits of the recommender GitHub repository is that it provides a set of options and shows which algorithms are best for solving certain types of problems. It also provides a rough framework for how to switch between different algorithms. If model performance accuracy isn’t enough, an algorithm better suited for real-time results is needed, or the originally chosen algorithm isn’t the best fit for the type of data being used, a data scientist may want to switch to a different algorithm.

Choosing, understanding, and implementing newer models for recommender systems can be costly

Selecting the right recommender algorithm from scratch and implementing new models for recommender systems can be costly as they require ample time for training and testing as well as large amounts of compute power. The recommender GitHub repository streamlines the selection process, reducing costs by saving data scientists time in testing many algorithms that are not a good fit for their projects/scenarios. This, coupled with Azure’s various pricing options, reduces data scientists’ costs on testing and organization’s costs in deployment.

Implementing more state-of-the-art algorithms can appear daunting

When asked to build a recommender system, data scientists will often turn to more commonly known algorithms to alleviate the time and costs needed to choose and test more state-of-the-art algorithms, even if these more advanced algorithms may be a better fit for the project/data set. The recommender GitHub repository provides a library of well-known and state-of-the-art recommender algorithms that best fit certain scenarios. It also provides best practices that, when followed, make implementing more state-of-the-art algorithms easier to approach.

Data scientists are unfamiliar with how to use Azure Machine Learning service to train, test, optimize, and deploy recommender algorithms

Finally, the recommender GitHub repository provides best practices for how to train, test, optimize, and deploy recommender models on Azure and Azure Machine Learning (Azure ML) service. In fact, there are several notebooks available on how to run the recommender algorithms in the repository on Azure ML service. Data scientists can also take any notebook that has already been created and submit it to Azure with minimal or no changes.

Azure ML can be used intensively across various notebooks for tasks relating to AI model development, such as:

Hyperparameter tuning
Tracking and monitoring metrics to enhance the model creation process
Scaling up and out on compute like DSVM and Azure ML Compute
Deploying a web service to Azure Kubernetes Service
Submitting pipelines

Learn more

Utilize the GitHub repository for your own recommender systems.

Learn more about the Azure Machine Learning service.

Get started with a free trial of Azure Machine Learning service.
Quelle: Azure

Quest powers Spotlight Cloud with Azure

This blog post was co-authored by Liz Yu (Marketing), Bryden Oliver (Architect), Iain Shepard (Senior Software Engineer) at Spotlight Cloud, and Deborah Chen (Program Manager), Sri Chintala (Program Manager) at Azure Cosmos DB.

 

Spotlight Cloud is the first built on Azure database performance monitoring solution focused on SQL Server customers. Leveraging the scalability, performance, global distribution, high-availability, and built-in security of Microsoft Azure Cosmos DB, Spotlight Cloud combines the best of the cloud with Quest Software’s engineering insights from years of building database performance management tools.

As a tool that delivers database insights that lead customers to higher availability, scalability, and faster resolution of their SQL solutions, Spotlight Cloud needed a database service that provided those exact requirements on the backend as well.

Using Azure Cosmos DB and Azure Functions, Quest was able to build a proof of concept within two months and deploy to production in less than eight months.

“Azure Cosmos DB will allow us to scale as our application scales. As we onboard more customers, we value the predictability in terms of performance, latency, and the availability we get from Azure Cosmos DB.”

– Patrick O’Keeffe, VP of Software Engineering, Quest Software

Spotlight Cloud requirements

The amount of data needed to support a business continually grows. As data scales, so does Spotlight Cloud, as it needs to analyze all that data. Quest’s developers knew they needed a highly available database service with the following requirements and at affordable cost:

Collect and store many different types of data and send it to an Azure-based storage service. The data comes from SQL Server DMVs, OS performance counter statistics, SQL plans, and other useful information. The data collected varies greatly in size (100 bytes to multiple megabytes) and shape.
Accept 1,200 operations/second on the data with the ability to continue to scale as more customers use Spotlight Cloud.
Query and return data to aid in the diagnosis and analysis of SQL Server performance problems quickly.

After a thorough evaluation of many products, Quest chose Azure Functions and Azure Cosmos DB as the backbone of their solution. Spotlight Cloud was able to leverage both Azure Function apps and Azure Cosmos DB to reduce cost, improve performance, and deliver a better service to their customers.

Solution

Part of the core data flow in Spotlight Cloud. Other technologies used, not shown, include Event Hub, Application Insights, Key Vault, Storage, DNS.

The core data processing flow within Spotlight Cloud is built on Azure Functions and Azure Cosmos DB. This technology stack provides Quest with the high scale and performance they need.

Scale

 

Ingest apps handle >1,000 sets of customer monitoring data per second. To support this, Azure Functions consumption plan auto-scales up to 100s of VMs automatically.

Azure Cosmos DB provides guaranteed throughput for database and containers, measured in Request Units / second (RU/s), and backed by SLAs. By estimating the required throughput of the workload and translating it to RU/s, Quest was able to achieve predictable throughput of reads and writes against Azure Cosmos DB at any scale.

Performance

 

Azure Cosmos DB handles the write and read operations for Spotlight’s data at < 60 milliseconds. This enables customers’ SQL Server data to be quickly ingested and available for analysis in near real time.

High availability

 

Azure Cosmos DB provides 99.999% high availability SLA for reads and writes, when using 2+ regions. Availability is crucial for Spotlight Cloud’s customers, as many are in the healthcare, retail, and financial services industries and cannot afford to experience any database downtime or performance degradation. In the event a failover is needed, Azure Cosmos DB does automatic failover with no manual intervention, enabling business continuity.

With turnkey global distribution, Azure Cosmos DB handles automatic and asynchronous replication of data between regions. To take full advantage of their provisioned throughput, Quest designated one region to handle writes (data ingest) and another for reads. As a result, users’ read response times are never impacted by the write volume.

Flexible schema

 

Azure Cosmos DB accepts JSON data of varying size and schema. This enabled Quest to store a variety of data from diverse sources, such as SQL Server DMVs, OS performance counter statistics, etc., and removed the need to worry about fixed schemas or schema management.

Developer productivity

 

Azure Functions tooling made the development and coding process very smooth, which enabled developers to be productive immediately. Developers also found Azure Cosmos DB’s SQL query language to be easy to use, reducing the ramp-up time.

Cost

 

The Azure Functions consumption pricing model charges only for the compute and memory each function invocation uses. Particularly for lower-volume microservices, this lets users operate at low cost. In addition, using Azure Functions on a consumption plan gives Quest the ability to have failover instances on standby at all times, and only incur cost if failover instances are actually used.

From a Total Cost of Ownership (TCO) perspective, Azure Cosmos DB and Azure Functions are both managed solutions, which reduced the amount of time spent on management and operations. This enabled the team to focus on building services that deliver direct value to their customers.

Support

Microsoft engineers are directly available to help with issues, provide guidance and share best practices

With Spotlight Cloud, Quest’s customers have the advantage of storing data in Azure instead of an on-premises SQL Server database. Customers also have access to all the analysis features that Quest provides in the cloud. For example, a customer can investigate the SQL workload and performance on their SQL Server in great detail to optimize the data and queries for their users – all powered by Spotlight Cloud running on top of Azure Cosmos DB.

"We were looking to upgrade our storage solution to better meet our business needs. Azure Cosmos DB gave us built-in high availability and low latency, which allowed us to improve our uptime and performance. I believe Azure Cosmos DB plays an important role in our Spotlight Cloud to enable customers to access real-time data fast."

– Efim Dimenstein, Chief Cloud Architect, Quest Software

Deployment Diagram of Spotlight Cloud’s Ingest and Egress app

Diagram above explained. Data is routed to an available ingest app by the Traffic Manager. The Ingest app writes data into the Azure Cosmos DB write region. Data consumers are routed via Traffic Manager to Egress app, which then reads data from the Azure Cosmos DB read region.

Learnings and best practices

In building Spotlight Cloud, Quest gained a deep understanding into how to use Azure Cosmos DB in the most effective way: 

 

Understand Azure Cosmos DB’s provisioned throughput model (RU/s)

 

Quest measured the cost of each operation, the number of operations/second, and provisioned the total amount of throughput required in Azure Cosmos DB.

Since Azure Cosmos DB cost is based on storage and provisioned throughput, choosing the right amount of RUs was key to using Azure Cosmos DB in a cost effective manner.

Choose a good partition strategy

 

Quest chose a partition key for their data that resulted in a balanced distribution of request volume and storage. This is critical because Azure Cosmos DB shards data horizontally and distributes total provisioned RUs evenly among the partitions of data.

During the development stage, Quest experimented with several choices of partition key and measured the impact on the performance. If a partition key strategy was unbalanced, a workload would require more RUs than with a balanced partition strategy.

Quest chose a synthetic partition key that incorporated Server Id and type of data being stored. This gave a high number of distinct values (high cardinality), leading to an even distribution of data – crucial for a write heavy workload.

Tune indexing policy

 

For Quest’s write-heavy workload, tuning index policy and RU cost on writes was key to achieving good performance. To do this, Quest modified the Azure Cosmos DB indexing policy to explicitly index commonly queried properties in a document and exclude the rest. In addition, Quest included only a few commonly used properties in the body of the document and encoded the rest of the data into a single property.

Scale up and down RUs based on data access pattern

 

In Spotlight Cloud, customers tend to access recent data more frequently than the older data. At the same time, new data continues to be written in a steady stream, making it a write-heavy workload.

To tune the overall provisioned RUs of the workload, Quest split the data into multiple containers. A new container is created regularly (e.g. every week to a few months) with high RUs, ready to receive writes.

Once the next new container is ready, the previous container’s RUs is reduced to only what is required to serve the expected read operations. Writes are then directed to the new container with high number of RUs.

Tour of Spotlight Cloud’s user interface

About Quest

Quest has provided software solutions for the fast paced world of enterprise IT since 1987. They are a global provider to 130,000 companies across 100 countries, including 95 percent of the Fortune 500 and 90% of the Global 1000.

Find out more about Spotlight Cloud on Twitter, Facebook, and LinkedIn.
Quelle: Azure