Azure Cost Management updates – June 2019

Whether you're a new student, thriving startup, or the largest enterprise, you have financial constraints and you need to know what you're spending, where, and how to plan for the future. Nobody wants a surprise when it comes to the bill, and this is where Azure Cost Management comes in.

We're always looking for ways to learn more about your challenges and how Cost Management can help you better understand where you're accruing costs in the cloud, identify and prevent bad spending patterns, and optimize costs to empower you to do more with less.

Here are the improvements that we'll be looking at today, all based on your feedback:

Reservation and marketplace purchases for Enterprise Agreements and AWS
Forecasting your Azure and AWS costs
Standardizing cost and usage terminology for Enterprise Agreements and Microst Customer Agreements
Keeping an eye on costs across subscriptions with management group budgets
Updating your dashboard tiles
Expanded availability of resource tags in cost reporting
The new Cost Management YouTube channel

Let's dig into the details.

 

Reservation and marketplace purchases for Enterprise Agreements and AWS

Effective cost management starts by getting all your costs into a single place with a single taxonomy. Now, with the addition of reservation and marketplace purchases, you have a more complete picture of your Enterprise Agreements (EA) for Azure and AWS costs, and can track large reservation costs back to the teams using the reservation benefit. Breaking reservation purchases down will simplify cost allocation efforts, making it easier than ever to manage internal chargeback.

Start by opening cost analysis and changing scope to your EA billing account, AWS consolidated account, or a management group which spans both. You'll notice four new grouping and filtering options to break down and drill into costs:

Charge type indicates which costs are from usage, purchases, and refunds.
Publisher type indicates which costs are from Azure, AWS, and marketplace. Marketplace costs include all clouds. Use Provider to distinguish between the total Azure and AWS costs, and first and third-party costs.
Reservation specifies what the reservation costs are associated with, if applicable.
Frequency indicates which costs are usage-based, one-time fees, or recurring charges.

By default, cost analysis shows your actual cost as it is on your bill. This is ideal for reconciling your invoice, but results in visible spikes from large purchases. This also means usage against a reservation will show no cost, since it was prepaid, and subscription and resource group readers won't have any visibility into their effective costs. This is where amortization comes in.

Switch to the amortized cost view to break down reservation purchases into daily chunks and spread them over the duration of the reservation term. As an example, instead of seeing a $365 purchase on January , you will see a $1 purchase every day from January 1 to December 31. In addition to basic amortization, these costs are also reallocated and associated with the specific resources which used the reservation. For example, if that $1 daily charge is split between two virtual machines, you'll see two $0.50 charges for the day. If part of the reservation is not utilized for the day, you'll see one $0.50 charge associated with the applicable virtual machine and another $0.50 charge with a new charge type titled UnusedReservation.

As an added bonus subscription, resource group, and AWS linked account readers can also see their effective costs by viewing amortized costs. They won't be able to see the purchases, which are only visible on the billing account, but they can see their discounted cost based on the reservation.

To build a simple chargeback report, switch to amortized cost, select no granularity to view the total costs for the period, group by resource group, and change to table view. Then, download the data to Excel or CSV for offline analysis or to merge with your own data.

If you need to automate getting cost data, you have two options. Use the Query API for rich analysis with dynamic filtering, grouping, and aggregation or use the UsageDetails API for the full, unaggregated cost and usage data. Note UsageDetails is only available for Azure scopes. The general availability (GA) version of these APIs is 2019-01-01, but you'll want to use 2019-04-01-preview to include reservation and Marketplace purchases.

As an example, let's get an aggregated view of amortized costs broken down by charge type, publisher type, resource group—left empty for purchases, and reservation—left empty if not applicable.

POST https://management.azure.com/{scope}/providers/Microsoft.CostManagement/query?api-version=2019-04-01-preview
Content-Type: application/json

{
"type": "AmortizedCost",
"timeframe": "Custom",
"timePeriod": { "from": "2019-06-01", "to": "2019-06-30" },
"dataset": {
"granularity": "None",
"aggregation": {
"totalCost": { "name": "PreTaxCost", "function": "Sum" }
},
"grouping": [
{ "type": "dimension", "name": "ChargeType" },
{ "type": "dimension", "name": "PublisherType" },
{ "type": "dimension", "name": "Frequency" },
{ "type": "dimension", "name": "ResourceGroup" },
{ "type": "dimension", "name": "SubscriptionName" },
{ "type": "dimension", "name": "SubscriptionId" },
{ "type": "dimension", "name": "ReservationName" },
{ "type": "dimension", "name": "ReservationId" }
]
}
}

And if you don't need the aggregation and prefer the full, raw dataset for Azure scopes:

GET https://management.azure.com/{scope}/providers/Microsoft.Consumption/usageDetails?metric=AmortizedCost&$filter=properties/usageStart+ge+'2019-06-01'+AND+properties/usageEnd+le+'2019-06-30'&api-version=2019-04-01-preview

If you need actual costs to show purchases as they are shown on your bill, simply change the type or metric to ActualCost. For more information about these APIs, refer to the Query and UsageDetails API documentation. The published docs show the GA version, but they both work the same for the 2019-04-01-preview API version outside of the new type/metric attribute.

Note that Cost Management APIs work across all scopes above resources. Namely, resource group, subscription, management group via Azure roll-based access control (RBAC) access, EA billing accounts (enrollments), departments, enrollment accounts via EA portal access, AWS consolidated, and linked accounts via Azure RBAC. To learn more about scopes, including how to determine your scope ID or manage access, see our documentation "Understand and work with scopes."

Support for reservation and marketplace purchases is currently available in preview in the Azure portal, but will roll out globally in the coming weeks. In the meantime, please check it out and let us know if you have any feedback.

 

Forecasting your Azure and AWS costs

History teaches us a lot, and knowing where you've been is critical to understanding where you're going. This is no less true when it comes to managing costs. You may start with historical costs to understand application and organization trends, but to really get into a healthy, optimized state, you need to plan for the future. Now you can with Cost Management forecasts.

Check your forecasted costs in cost analysis to anticipate and visualize cost trends, and proactively take action to avoid budget or credit overages on any scope. From a single application in a resource group, to the entire subscription or billing account, to higher-level management groups spanning both Azure and AWS resources. Learn about connecting your AWS account in last month's wrap up here.

Cost Management forecasts are in preview in the Azure portal, and will roll out globally in the coming weeks. Check it out and let us know what you'd like to see next.

 

Standardizing cost and usage terminology for Enterprise Agreement and Microsoft Customer Agreement

Depending on whether you use a pay-as-you-go (PAYG), Enterprise Agreement (EA), Cloud Solution Provider (CSP), or Microsoft Customer Agreement (MCA) account, you may be used to a different terminology. These differences are minor and won't impact your ability to understand and break down your bills, but they do introduce a challenge as your organization grows and needs a more holistic cost management solution, spanning multiple account types. With the addition of AWS and eventual migration of PAYG, EA, and CSP accounts into MCA, this becomes even more important. In an effort to streamline the transition to MCA at your next EA renewal, Cost Management now uses new column or property names to align to MCA terminology. Here are the primary differences you can expect to see for EA accounts:

EnrollmentNumber → BillingAccountId/BillingProfileId

​EA enrollments are represented as "billing accounts" within the Azure portal today, and they will continue to be mapped to a BillingAccountId within the cost and usage data. No change there. MCA also introduces the ability to create multiple invoices within a billing account. The configuration of these invoices is called a "billing profile". Since EA can only have a single invoice, the enrollment effectively maps to a billing profile. In line with that conceptual model, the enrollment number will be available as both a BillingAccountId and BillingProfileId.

DepartmentName → InvoiceSectionName

​MCA has a concept similar to EA departments, which allows you to group subscriptions within the invoice. These are called "invoice sections" and are nested under a billing profile. While the EA invoice isn't changing as part of this effort, EA departments will be shown as InvoiceSectionName within the cost data for consistency.

ProductOrderName (new)

​New property to identify the larger product the charge applies to, like the Azure subscription offer.

PublisherName (new)

​New property to indicate the publisher of the offering.

ServiceFamily (new)

​New property to group related meter categories.

Organizations looking to renew their EA enrollment into a new MCA should strongly consider moving from the key-based EA APIs (such as consumption.azure.com) to the latest UsageDetails API (version 2019-04-01-preview) based on these new properties to minimize future migration work. The key-based APIs are not supported for MCA billing accounts.

To learn more about the new terminology, see our documentation "Understand the terms in your Azure usage and charges file."

 

Keeping an eye on costs across subscriptions with management group budgets

Every organization has a bottom line. Cost Management budgets help you make sure you don't hit yours. And now, you can create budgets that span both Azure and AWS resources using management groups.

Organize subscriptions into management groups, and use filters to perfectly tune the budget that's right for your teams.

To learn more, see our tutorial "Create and manage budgets."

 

Updating your dashboard tiles

You already know you can pin customized views of cost analysis to the dashboard.

You may have noticed these tiles were locked to the specific date range you selected when pinning it. For instance, if you chose to view this month's costs in January, the tile would always show January, even in February, March, and so on. This is no longer the case.

Cost analysis tiles now maintain the built-in range you selected in the date picker. If you pin "this month," you'll always get the current calendar month. If you pin "last 7 days," you'll get a rolling view of the last 7 days. If you select a custom date range, however, the tile will always show that specific date range.

To get the updated behavior, please update your pinned tiles. Simply click the chart on the tile to open cost analysis, select the desired date range, and pin it back to the dashboard. Your new tile will always keep the exact view you selected.

What else would help you build out your cost dashboard? Do you need other date ranges? Let us know.

 

Expanded availability of resource tags in cost reporting

Tagging is the best way to organize and categorize your resources outside of the built-in management group, subscription, and resource group hierarchy, allowing you to add your own metadata and build custom reports using cost analysis. While most Azure resources support tags, some resource types do not. Here are the latest resource types which now support tags:

App Service environments
Data Factory services
Event Hub namespaces
Load balancers
Service Bus namespaces

Remember tags are a part of every usage record and are only available in Cost Management reporting after the tag is applied. Historical costs are not tagged, so update your resources today for the best cost reporting.

 

The new Cost Management YouTube channel

Last month, we talked about eight new quickstart videos to get you up and running with Cost Management quickly. Subscribe to the new Azure Cost Management YouTube channel to stay in the loop with new videos as they're released. Here's the newest video in our cost optimization collection:

Five tips to help you save money and manage costs with Azure

Let us know what other topics you'd like to see covered.

 

What's next?

These are just a few of the big updates from the last month. We're always listening and making constant improvements based on your feedback, so please keep the feedback coming! 

Follow @AzureCostMgmt on Twitter and subscribe to the YouTube channel for updates, tips, and tricks. And, as always, share your ideas and vote up others in the Cost Management feedback forum.
Quelle: Azure

Azure. Source–Volume 89

Dear Azure fans, Azure.Source is going on hiatus. Thank you for reading each week and be sure to follow @Azure for updates and new ways to learn more.

Now available

Announcing the general availability of Azure premium files

We are excited to announce the general availability of Azure premium files for customers optimizing their cloud-based file shares on Azure. Premium files offers a higher level of performance built on solid-state drives (SSD) for fully managed file services in Azure.

Premium tier is optimized to deliver consistent performance for IO-intensive workloads that require high-throughput and low latency. Premium file shares store data on the latest SSDs, making them suitable for a wide variety of workloads like databases, persistent volumes for containers, home directories, content and collaboration repositories, media and analytics, high variable and batch workloads, and enterprise applications that are performance sensitive. Our existing standard tier continues to provide reliable performance at a low cost for workloads less sensitive to performance variability, and is well-suited for general purpose file storage, development/test, backups, and applications that do not require low latency.

Leveraging complex data to build advanced search applications with Azure Search

Data is rarely simple. Not every piece of data we have can fit nicely into a single Excel worksheet of rows and columns. Data has many diverse relationships, such as the multiple locations and phone numbers for a single customer .or multiple authors and genres of a single book. Of course, relationships typically are even more complex than this, and as we start to leverage AI to understand our data the additional learnings we get only add to the complexity of relationships. For that reason, expecting customers to have to flatten the data so it can be searched and explored is often unrealistic. We heard this often and it quickly became our number one most requested Azure Search feature. Because of this we were excited to announce the general availability of complex types support in Azure Search. In this post, we explain what complex types adds to Azure Search and the kinds of things you can build using this capability.

Azure Blockchain Workbench 1.7.0 integration with Azure Blockchain Service

The release of Microsoft Azure Blockchain Workbench 1.7.0, which along with our new Azure Blockchain Service, can further enhance your blockchain development and projects. You can deploy a new instance of Blockchain Workbench through the Azure portal or upgrade your existing deployments to 1.7.0 using the upgrade script. This update includes the improvements such as integration with Azure Blockchain Service, and enhance compatibility with Quorum.

New PCI DSS Azure Blueprint makes compliance simpler

Announcing our second Azure Blueprint for an important compliance standard with the release of the PCI-DSS v3.2.1 blueprint. The new blueprint maps a core set of policies for Payment Card Industry (PCI) Data Security Standards (DSS) compliance to any Azure deployed architecture, allowing businesses such as retailers to quickly create new environments with compliance built in to the Azure infrastructure. Azure Blueprints is a free service that enables customers to define a repeatable set of Azure resources that implement and adhere to standards, patterns, and requirements. Azure Blueprints allow customers to set up governed Azure environments that can scale to support production implementations for large-scale migrations.

Now in preview

Event-driven analytics with Azure Data Lake Storage Gen2

Announcing that Azure Data Lake Storage Gen2 integration with Azure Event Grid is in preview. This means that Azure Data Lake Storage Gen2 can now generate events that can be consumed by Event Grid and routed to subscribers with webhooks, Azure Event Hubs, Azure Functions, and Logic Apps as endpoints. With this capability, individual changes to files and directories in Azure Data Lake Storage Gen2 can automatically be captured and made available to data engineers for creating rich big data analytics platforms that use event-driven architectures.

Technical content

How to deploy your machine learning models with Azure Machine Learning

Azure Machine Learning service is a cloud service that you use to train, deploy, automate, and manage machine learning models, all at the broad scale that the cloud provides. The service fully supports open-source technologies such as PyTorch, TensorFlow, and scikit-learn and can be used for any kind of machine learning, from classical ml to deep learning, supervised and unsupervised learning. In this article you will learn to deploy your machine learning models with Azure Machine Learning.

Azure Cloud Shell Tips for SysAdmins Part II – Using the Cloud Shell tools to Migrate

In the last blog post Azure Cloud Shell Tips for SysAdmins (bash) the author discussed some of the tools that the Azure Cloud Shell for bash already has built into it.  This time he goes deeper and show you how to utilize a combination of the tools to create an UbuntuLTS Linux server.  Once the server is provisioned, he will demonstrate how to use Ansible to deploy Node.js from the nodesource binary repository.

Step-By-Step: Migrating The Active Directory Certificate Service From Windows Server 2008 R2 to 2019

End of support for Windows Server 2008 R2 has been slated by Microsoft for January 14th 2020.  Said announcement increased interest in a previous post detailing steps on Active Directory Certificate Service migration from server versions older than 2008 R2.  Many subscribers of ITOpsTalk.com have reached out asking for an update of the steps to reflect Active Directory Certificate Service migration from 2008 R2 to 2016 / 2019 and of course our team is happy to oblige.

Home Grown IoT – Local Dev

Now that we’re starting to build our IoT application it’s time to start talking about the local development experience for the application. At the end of the day I use IoT Edge to do the deployment onto the device and manage the communication with IoT Hub and there is a very comprehensive development guide for Visual Studio Code and Visual Studio 2019. The workflow of this is to create a new IoT Edge project, setup IoT Edge on your machine and do deployments to it that way. This is the way I’d recommend going about it yourself as it gives you the best replication of production and local development.

Delivering static content via Azure CDN | Azure Friday

In one of the prior episodes we learned how to serve a static website from Azure's blob storage<?XML:NAMESPACE PREFIX = "[default] http://www.w3.org/2000/svg" NS = "http://www.w3.org/2000/svg" /> . This is great for a low volume web site. As your site starts getting more hits, you wanted to deliver the content closer to the end user. In this episode, we will learn how to deliver a static content via Azure Content Delivery Network(CDN). Azure CDN offers developers a global solution for rapidly delivering high-bandwidth content to users by caching their content at strategically placed physical nodes across the world.

Azure shows

Deploy your web app in Windows containers on Azure App Service | Azure Friday

Windows Container support is available in preview in Azure App Service. By deploying applications via Windows Containers in Azure App Service you can install your dependencies inside the container, call APIs currently blocked by the Azure App Service sandbox and use the power of containers to migrate applications for which you no longer have the source code. All of this and you still get to use the awesome feature set enabled by Azure App Service such as auto-scale, deployment slots and increased developer productivity.

Using open data to build family trees | The Open Source Show

Erica Joy joins Ashley McNamara to share her not-so-secret personal mission: making genealogy information open, queryable, and easily parsable. She shares a bit about why this is so critical, common challenges, and tips for re-building your own family tree – or using open data to uncover whatever the information you need for your personal mission.

Supporting Windows forms and WPF in .NET Core 3 | On .NET

There is significant effort happening to add support for running desktop applications on .NET Core 3.0. In this episode, Jeremy interviews Mike Harsh about some of the work being done and decisions being made to enable Windows Forms and WPF applications to run well on .NET Core 3.0 and beyond.

Five things about RxJS and reactive programming | Five Things

Where do RxJS, Reactive Programming and the Redux pattern fit into your developer workflow? Where can you learn form the community leaders? Does wearing a hoodie make you a better developer? Oh and remember, go to RxJS Live and drinks are on Aaron!

How to use the Global Search in the Azure portal | Azure Portal Series

In this video of the Azure Portal “How To” Series, you will learn how to find Azure services, resources, documentation, and more using the Global Search in the Azure portal.

Episode 285 – The Azure Journey | The Azure Podcast

Sujit, Kendall, and Cynthia talk with the one and only Richard Campbell on how to tell the cloud story, the conversations to have with customers as they enter the cloud and the implications of globally distributed cloud that needs to be considered. Probably one of our favorite shows.

HTML5 audio not supported

Industries and partners

Solving the problem of duplicate records in healthcare

As the U.S. healthcare system continues to transition away from paper to more a digitized ecosystem, the ability to link an individual’s medical data together correctly becomes increasingly challenging. Patients move, marry, divorce, change names and visit multiple providers throughout their lifetime, with each visit creating new records, and the potential for inconsistent or duplicate information grows. Duplicate medical records often occur as a result of multiple name variations, data entry errors, and lack of interoperability—or communication—between systems. Poor patient identification and duplicate records in turn lead to diagnosis errors, redundant medical tests, skewed reporting and analytics, and billing inaccuracies. The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we will describe how one Microsoft partner, Nextgate, uses Azure to solve a unique problem.

A solution to manage policy administration from end to end

Legacy systems can be a nightmare for any business to maintain. In the insurance industry, carriers struggle not only to maintain these systems but to modify and extend them to support new business initiatives. The insurance business is complex, every state and nation has its own unique set of rules, regulations, and demographics. Creating new products such as an automobile policy has traditionally required the coordination of many different processes, systems, and people. These monolithic systems traditionally used to create new products are inflexible and creating a new product can be an expensive proposition. The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we describe how one Microsoft partner, Sunlight Solutions, uses Azure to solve a unique problem.

Using natural language processing to manage healthcare records

The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we describe how SyTrue, a Microsoft partner focusing on healthcare uses Azure to empower healthcare organizations to improve efficiency, reduce costs, and improve patient outcomes.

Azure Cosmos DB: A competitive advantage for healthcare ISVs

CitiusTech is a specialist provider of healthcare technology services which helps its customers to accelerate innovation in healthcare. CitiusTech used Azure Cosmos DB to simplify the real-time collection and movement of healthcare data from variety of sources in a secured manner. With the proliferation of patient information from established and current sources, accompanied with scrupulous regulations, healthcare systems today are gradually shifting towards near real-time data integration.
Quelle: Azure

Helping move healthcare organizations to Azure

Today’s healthcare organizations are expected to be agile, reduce costs, and direct capital toward revenue generating activities that improve patient outcomes. The cloud is a key part of the answer, but implementing a new solution on the cloud also requires new skills especially around governance, compliance with HIPAA, and security practices. Many healthcare organizations look to an experienced partner to help them migrate solutions from on-premises to the cloud, while building in the right set of structures to seamlessly handle known and future challenges.

The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we describe how one Microsoft partner uses Azure to solve a unique problem.

Wanted: Governance and compliance expertise

For organizations that have moved to the cloud, a lack of governance and understanding about the way cloud services work can lead to wasted spending, unpredictable cloud service bills, and cloud vendor lock-in. The rapid growth of cloud infrastructures also creates a dizzying array of possibilities that can keep a team uncertain of the correct path and second guessing their choices, which can lead to delay and add risk of failure.

Now, healthcare CIOs increasingly rely on cloud platforms, but they run into new problems. To prevent the inevitable difficulties requires a staff that is fully enabled with the right skills for compliance, privacy, and security. Health IT professionals need guidance on how to move an on-premises healthcare infrastructure to a cloud platform, and ensure HIPAA compliance, policies, safeguards, and resources are in place.

Here are the major areas that require thought and planning:

Privacy, compliance concerns: Protecting patient data is a persistent concern, along with implementation, uncertainty, and risk. Concerns about HIPAA compliance, cloud, and legacy system integration are among the major obstacles that have kept healthcare IT on-premises.
Budget constraints, cost optimization: Cloud service bills are often highly detailed and complicated, making it difficult to determine which application, department, or resource is the source of a cost overrun.
Technical hurdles: Healthcare IT professionals may not have the skills or resources to leverage cloud services to do things like extend an on-premises datacenter to a hybrid cloud.
Training: Retaining and enabling IT staff is a key challenge, and education on any new solution is critical to success. Everyone should have easy to understand resources regardless of the role whether it be IT leaders, administrators, developers, and/or database administrators.
Gaps in capabilities: Even with an on-premises solution, many use special services from a vendor. Planning should include those partners as well as specialized areas that the vendors don’t currently address.

Solution

Burwood Group is a Microsoft partner that specializes in moving healthcare organizations to Azure. If a client has a secure, on-premises network, Burwood will build a secure cloud network and leverage the same regulatory controls used for an on-premises installations. They will also educate technology teams on endpoint security and serverless security, with emphasis on HIPAA compliance in the cloud.

The consulting firm offers extensive training. For example, through a one-day class, they provide the basic education to have a successful implementation in Azure, with an emphasis on healthcare requirements in the cloud. This workshop includes hands-on lab exercises and is 100 percent focused on pertinent, practical, and actionable knowledge.

Benefits

Standardization: As a cloud team, nothing is left to guess work. Instead, consistency is instilled across the team. Through education, Burwood introduces the healthcare datacenter in Azure.
Flexibility: IT teams may need to work with multiple cloud architectures for healthcare. This occurs as care is increasingly managed across settings with more interoperability across applications and business entities. Understanding best practices for the cloud allows expertise that is independent of any application or vendor.
Control: When it comes to cloud governance for healthcare, organizations need to control cloud sprawl. As personnel enter or leave an organization, permissions must be carefully allowed or revoked to prevent security breaches. Burwood provides education on these subjects: What is going into and out of Azure? Who has rights to resources in Azure? These types of questions are answered.
Service catalog: Burwood seeks to keep users informed of new services through a service catalog. Users are instructed about the following.

Handling cloud service requests and change management.
Expanding the current service catalog through an Azure for healthcare IT emphasis.
Potential items that users can request through the service catalog in Azure.

Indexing: All resources in the cloud must be tagged with cost center, creation date, and more.
IP awareness: Users are instructed to be very careful of public IP address assignments, and the potential of creating vulnerabilities.

Services

The company has a proficiency in both healthcare and Azure technology. These are a few of the Azure services used to create custom solutions:

Azure portal
Azure Resource Manager
Azure role based access control
Azure Active Directory
Azure Load Balancer

Next steps

To learn more about other industry solutions, go to the Azure for healthcare page. To find more details about consulting and a one day Azure University for healthcare workshop, go to the Azure Marketplace listing for the Burwood Group and select Contact me.
Quelle: Azure

Leveraging complex data to build advanced search applications with Azure Search

Data is rarely simple. Not every piece of data we have can fit nicely into a single Excel worksheet of rows and columns. Data has many diverse relationships such as the multiple locations and phone numbers for a single customer or multiple authors and genres of a single book. Of course, relationships typically are even more complex than this, and as we start to leverage AI to understand our data the additional learnings we get only add to the complexity of relationships. For that reason, expecting customers to have to flatten the data so it can be searched and explored is often unrealistic. We heard this often and it quickly became our number one most requested Azure Search feature. Because of this we were excited to announce the general availability of complex types support in Azure Search. In this post, I want to take some time to explain what complex types adds to Azure Search and the kinds of things you can build using this capability. 

Azure Search is a platform as a service that helps developers create their own cloud search solutions.

What is complex data?

Complex data consists of data that includes hierarchical or nested substructures that do not break down neatly into a tabular rowset. For example a book with multiple authors, where each author can have multiple attributes, can’t be represented as a single row of data unless there is a way to model the authors as a collection of objects. Complex types provide this capability, and they can be used when the data cannot be modeled in simple field structures such as strings or integers.

Complex types applicability

At Microsoft Build 2019,  we demonstrated how complex types could be leveraged to build out an effective search application. In the session we looked at the Travel Stack Exchange site, one of the many online communities supported by StackExchange.

The StackExchange data was modeled in a JSON structure to allow easy ingestion it into Azure Search. If we look at the first post made to this site and focus on the first few fields, we see that all of them can be modeled using simple datatypes, including tags which can be modeled as a collection, or array of strings.

{
"id": "1",
"CreationDate": "2011-06-21T20:19:34.73",
"Score": 8,
"ViewCount": 462,
"BodyHTML": "<p>My fiancée and I are looking for a good Caribbean cruise in October and were wondering which
"Body": "my fiancée and i are looking for a good caribbean cruise in october and were wondering which islands
"OwnerUserId": 9,
"LastEditorUserId": 101,
"LastEditDate": "2011-12-28T21:36:43.91",
"LastActivityDate": "2012-05-24T14:52:14.76",
"Title": "What are some Caribbean cruises for October?",
"Tags": [
"caribbean",
"cruising",
"vacations"
],
"AnswerCount": 4,
"CommentCount": 4,
"CloseDate": "0001-01-01T00:00:00",​

However, as we look further down this dataset we see that the data quickly gets more complex and cannot be mapped into a flat structure. For example, there can be numerous comments and answers associated with a single document.  Even votes is defined here as a complex type (although technically it could have been flattened, but that would add work to transform the data).

"CloseDate": "0001-01-01T00:00:00",
"Comments": [
{
"Score": 0,
"Text": "To help with the cruise line question: Where are you located? My wife and I live in New Orlea
"CreationDate": "2011-06-21T20:25:14.257",
"UserId": 12
},
{
"Score": 0,
"Text": "Toronto, Ontario. We can fly out of anywhere though.",
"CreationDate": "2011-06-21T20:27:35.3",
"UserId": 9
},
{
"Score": 3,
"Text": ""Best" for what? Please read [this page](http://travel.stackexchange.com/questions/how-to
"UserId": 20
},
{
"Score": 2,
"Text": "What do you want out of a cruise? To relax on a boat? To visit islands? Culture? Adventure?
"CreationDate": "2011-06-24T05:07:16.643",
"UserId": 65
}
],
"Votes": {
"UpVotes": 10,
"DownVotes": 2
},
"Answers": [
{
"IsAcceptedAnswer": "True",
"Body": "This is less than an answer, but more than a comment…nnA large percentage of your travel b
"Score": 7,
"CreationDate": "2011-06-24T05:12:01.133",
"OwnerUserId": 74

All of this data is important to the search experience. For example, you might want to:

Search for and highlight phrases not only in the original question, but also in any of the comments.
Limit documents to those where an answer was provided by a specific user.
Boost certain documents higher in the search results when they have a higher number of up votes.

In fact, we could even improve on the existing StackExchange search interface by leveraging Cognitive Search to extract key phrases from the answers to supply potential phrases for autocomplete as the user types in the search box.

All of this is now possible because not only can you map this data to a complex structure, but the search queries can support this enhanced structure to help build out a better search experience.

Next Steps

If you would like to learn more about Azure Search complex types, please visit the documentation, or check out the video and associated code I made which digs into this Travel StackExchange data in more detail.
Quelle: Azure

Azure Blockchain Workbench 1.7.0 integration with Azure Blockchain Service

We’re excited to share the release of Microsoft Azure Blockchain Workbench 1.7.0, which along with our new Azure Blockchain Service, can further enhance your blockchain development and projects. You can deploy a new instance of Blockchain Workbench through the Azure portal or upgrade your existing deployments to 1.7.0 using the upgrade script. 

This update includes the following improvements:

Integration with Azure Blockchain Service

With the Azure Blockchain Service now in preview, you can develop directly with Blockchain Workbench on Azure Blockchain Service as the underlying blockchain. For those of you who have been on this blockchain journey with Microsoft, there are now templates in Azure which make it faster to configure and deploy a private blockchain network, but it’s still up to you to maintain and run your blockchain nodes, including upgrading to new versions, installing security patches, and more. Azure Blockchain Service simplifies the maintenance of the underlying blockchain network by running a fully managed blockchain node for you.

 

 

Blockchain Workbench helps with building the scaffolding needed on top of a blockchain network to quickly iterate and develop blockchain solutions. Workbench 1.7.0 enables you to easily deploy the Azure Blockchain Service directly with Workbench. To deploy Workbench from the Azure Marketplace, navigate to the Advanced settings blade and select Create new blockchain network under Blockchain settings.

 

Selecting this option will automatically deploy an Azure Blockchain Service node for you. Note that if you rotate the primary API key on the primary transaction node on your Azure Blockchain Service, you need to change the key of the configured RPC endpoint on Blockchain Workbench. Update the Key Vault with the new key and reboot the VMs.

Enhanced compatibility with Quorum

One of the highly requested features from customers is adding compatibility for additional blockchain network protocols. In previous releases of Blockchain Workbench, the default blockchain network that is configured is an Ethereum Proof-of-Authority (PoA) blockchain network. With Blockchain Workbench 1.7.0, we have added compatibility with the Quorum blockchain network.

For customers who are looking to build blockchain applications on top of Quorum, you can now develop and build your Quorum based applications directly with Blockchain Workbench.

You can stay up to date on Azure Blockchain Service by following the team on Twitter @MSFTBlockchain. Please use the Blockchain UserVoice to provide feedback and suggest features and ideas. Your input is helping make this a great service. We look forward to hearing from you.
Quelle: Azure

A solution to manage policy administration from end to end

Legacy systems can be a nightmare for any business to maintain. In the insurance industry, carriers struggle not only to maintain these systems but to modify and extend them to support new business initiatives. The insurance business is complex, every state and nation has its own unique set of rules, regulations, and demographics. Creating new products such as an automobile policy has traditionally required the coordination of many different processes, systems, and people. These monolithic systems traditionally used to create new products are inflexible and creating a new product can be an expensive proposition.

The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we describe how one Microsoft partner, Sunlight Solutions, uses Azure to solve a unique problem.

Monolithic systems and their problems

Insurers have long been restricted by complex digital ecosystems created by single-service solutions. Those tasked with maintaining such legacy, monolithic systems struggle as the system ages and becomes more unwieldy. Upgrades and enhancements often require significant new development, large teams, and long-term planning which are expensive, unrealistic, and a drain on morale. Worse, they restrict businesses from pursuing new and exciting opportunities.

A flexible but dedicated solution

An alternative is a single solution provider that is well versed in the insurance business but able to create a dedicated and flexible solution, one that overcomes the problems of a monolith. Sunlight is such a provider. It allows insurance carriers to leverage the benefits of receiving end-to-end insurance administration functionality from a single vendor. At the same time, their solution provides greater flexibility, speed-to-market, and fewer relationships to manage with lower integration costs.

Sunlight’s solution is a single system which manages end-to-end functionality across policy, billing, claims, forms management, customer/producer CRM, reporting and much more. According to Sunlight:

“We are highly flexible, managed through configuration rather than development. This allows for rapid speed to market for the initial deployment and complete flexibility when you need to make changes or support new business initiatives. Our efficient host and continuous delivery models address many of the industry’s largest challenges with respect to managing the cost and time associated with implementation, upgrades, and product maintenance.”

In order to achieve their goals of being quick but pliable, the architecture of the solution is literally a mixture of static and dynamic components. Static components are fields that do not change. Dynamic components such as lists populate at run time. This is conveyed in the graphic below, the solution uses static elements but lets users configure with dynamic parts as needed. The result is a faster cycle that maintains familiarity but allows a variety of data types.

In the figure above, data appears depending on the product. When products are acquired, for example through mergers, the static data can be mapped. If a tab exists for the product, it appears. For example, “benefits” and “deductibles” are not a part of every product.

Benefits

In brief, here are the key gains made by using Sunlight:

End-to-end functionality: Supports all products/coverages/lines of business
Cloud-based and accessible anywhere
Supports multiple languages and currencies
Globally configurable for international taxes and regional regulatory controls
Highly configurable by non-IT personnel
Reasonable price-point

Azure services

Azure Virtual Machines are used to implement the entire project life cycle quickly.
Azure Security Center provides a complete and dynamic infrastructure that continuously improves on its own.
Azure Site Recovery plans are simple to implement for our production layer.
Azure Functions is utilized in order to quickly replicate environments.
Azure Storage is used to keep the application light with a range of storage options for increased access time based on the storage type.

Next steps

To learn more about other industry solutions, go to the Azure for insurance page. To find more details about this solution, go to Sunlight Enterprise on the Azure Marketplace and select Contact me.
Quelle: Azure

New PCI DSS Azure Blueprint makes compliance simpler

I’m excited to announce our second Azure Blueprint for an important compliance standard with the release of the PCI-DSS v3.2.1 blueprint. The new blueprint maps a core set of policies for Payment Card Industry (PCI) Data Security Standards (DSS) compliance to any Azure deployed architecture, allowing businesses such as retailers to quickly create new environments with compliance built in to the Azure infrastructure.

Azure Blueprints is a free service that enables customers to define a repeatable set of Azure resources that implement and adhere to standards, patterns, and requirements. Azure Blueprints allow customers to set up governed Azure environments that can scale to support production implementations for large-scale migrations.

Azure Blueprints is another reason why Azure is a strong platform for compliance, with the industry’s broadest and deepest portfolio of 91 compliance offerings. Azure is built using some of the most rigorous security and compliance standards in the world, and includes multi-layered security provided by Microsoft across physical datacenters, infrastructure, and operations. Azure is also built for the specific compliance needs of key industries, including over 50 compliance offerings specifically for the retail, health, government, finance, education, manufacturing, and media industries.

Compliance with regulations and standards such as ISO 27001, FedRAMP and SOC is increasingly necessary for all types of organizations, making control mappings to compliance standards a natural application for Azure Blueprints. Azure customers, particularly those in regulated industries, have expressed strong interest in compliance blueprints to help ease their compliance burdens. In March, we announced the ISO 27001 Shared Services blueprint sample which maps a set of foundational Azure infrastructure, such as virtual networks and policies, to specific ISO controls.

The PCI DSS is a global information security standard designed to prevent fraud through increased control of credit card data. Organizations that accept payments from credit cards must follow PCI DSS standards if they accept payment cards from the five major credit card brands. Compliance with PCI DSS is also required for any organization that stores, processes, or transmits payment and cardholder data.

The PCI-DSS v3.2.1 blueprint includes mappings to important PCI DSS controls, including:

Segregation of duties. Manage subscription owner permissions.
Access to networks and network services. Implement role-based access control (RBAC) to manage who has access to Azure resources.
Management of secret authentication information of users. Audit accounts that don't have multi-factor authentication enabled.
Review of user access rights. Audit accounts that should be prioritized for review, including depreciated accounts and external accounts with elevated permissions.
Removal or adjustment of access rights. Audit deprecated accounts with owner permissions on a subscription.
Secure log-on procedures. Audit accounts that don't have multi-factor authentication enabled.
Password management system. Enforce strong passwords.
Policy on the use of cryptographic controls. Enforce specific cryptographic controls and audit use of weak cryptographic settings.
Event and operator logging. Diagnostic logs provide insight into operations that were performed within Azure resources.
Administrator and operator logs. Ensure system events are logged.
Management of technical vulnerabilities. Monitor missing system updates, operating system vulnerabilities, SQL vulnerabilities, and virtual machine vulnerabilities in Azure Security Center.
Network controls. Manage and control networks and monitor network security groups with permissive rules.
Information transfer policies and procedures. Ensure information transfer with Azure services is secure.

We are committed to helping our customers leverage Azure in a secure and compliant manner. Over the next few months we will release new built-in blueprints for HITRUST, UK National Health Service (NHS) Information Governance (IG) Toolkit, FedRAMP, and Center for Internet Security (CIS) Benchmark. If you would like to participate in any early previews please sign up with this form, or if you have a suggestion for a compliance blueprint, please share it via the Azure Governance Feedback Forum.

Learn more about the Azure PCI-DSS v3.2.1 blueprint in our documentation.
Quelle: Azure

Solving the problem of duplicate records in healthcare

As the U.S. healthcare system continues to transition away from paper to more a digitized ecosystem, the ability to link all of an individual’s medical data together correctly becomes increasingly challenging. Patients move, marry, divorce, change names and visit multiple providers throughout their lifetime, with each visit creating new records, and the potential for inconsistent or duplicate information grows. Duplicate medical records often occur as a result of multiple name variations, data entry errors, and lack of interoperability—or communication—between systems. Poor patient identification and duplicate records in turn lead to diagnosis errors, redundant medical tests, skewed reporting and analytics, and billing inaccuracies.

The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we will describe how one Microsoft partner, Nextgate, uses Azure to solve a unique problem.

Patient matching

The process of reconciling electronic health records is called “patient matching,” and it is a major obstacle to improving the quality of care coordination, and patient safety. Further, duplicate records are financially crippling, costing the average hospital $1.5 million and our nation’s healthcare system over $6 billion annually. As data sharing matures and the industry pivots toward value, an enterprise view of patient information is essential for informed clinical-decision making, effective episodic care, and a seamless patient-provider experience during every encounter.

As more data is generated and more applications are introduced into the health IT environment, today’s organizations must engage in more comprehensive patient matching approaches.

The puzzle of disjointed electronic health records

While electronic health records (EHRs) have become commonplace, the disjointed, competitive nature of IT systems contributes to a proliferation of siloed, disconnected information. Many EHR systems make sharing data arduous, even in a single-system electronic medical record environment. Further, master patient indexes (MPI) within EHR systems were designed for a single vendor-based environment and lack the sophisticated algorithms for linking data across various settings of care and disparate systems. When sent downstream, duplicate and disjointed patient demographics trigger further harm including increased waste and inefficiencies, suboptimal outcomes, and lost revenue. Without common technical standards in place, EHR systems continue to collect information in various formats that only serve to exacerbate the issue of duplicate record creation.

Solution

NextGate’s Enterprise Master Patient Index (EMPI) platform is a significant step towards improving a health system’s data management and governance framework. This solution manages patient identities for more than two-thirds of the U.S. population, and one-third of the U.K. population. It empowers clinicians and their organizations to make informed, life-saving decisions by seamlessly linking medical records from any given system and reconciling data discrepancies across multiple sites of care. The automated identity matching platform uses both probabilistic and deterministic matching algorithms to account for minor variations in patient data to generate a single best record that follows the patient throughout the care journey.

Benefits

Enhanced clinical decision-making.
Improved patient safety (or reduced medical errors.)
Decreased number of unnecessary or duplicate testing/procedures.
Improved interoperability and data exchange.
Trusted and reliable data quality.
Reduced number of denied claims and other reimbursement delays.
Improved administrative efficiencies.
Higher patient and provider satisfaction.

Azure services

Azure Security Center reinforces the security posture of the NextGate solution against threats, and provides recommendations to harden the security.
Azure Monitor provides telemetry data about the NextGate application to ensure its health.
Azure Virtual Machines provide compute power; enabling auto-scaling and supporting Linux and open source services
Azure SQL Database and Azure Database for PostgreSQL enable NextGate solutions to easily scale with more compute power (scale-up) or more database units (scale-out.)

Next steps

To find out more about this solution, go to Nextgate EMPI and click Contact me.
To see more about Azure in the healthcare industry see Azure for health.

Quelle: Azure

Event-driven analytics with Azure Data Lake Storage Gen2

Most modern-day businesses employ analytics pipelines for real-time and batch processing. A common characteristic of these pipelines is that data arrives at irregular intervals from diverse sources. This adds complexity in terms of having to orchestrate the pipeline such that data gets processed in a timely fashion.

The answer to these challenges lies in coming up with a decoupled event-driven pipeline using serverless components that responds to changes in data as they occur.

An integral part of any analytics pipeline is the data lake. Azure Data Lake Storage Gen2 provides secure, cost effective, and scalable storage for the structured, semi-structured, and unstructured data arriving from diverse sources. Azure Data Lake Storage Gen2’s performance, global availability, and partner ecosystem make it the platform of choice for analytics customers and partners around the world. Next comes the event processing aspect. With Azure Event Grid, a fully managed event routing service, Azure Functions, a serverless compute engine, and Azure Logic Apps, a serverless workflow orchestration engine, it is easy to perform event-based processing and workflows responding to the events in real-time.

Today, we’re very excited to announce that Azure Data Lake Storage Gen2 integration with Azure Event Grid is in preview! This means that Azure Data Lake Storage Gen2 can now generate events that can be consumed by Event Grid and routed to subscribers with webhooks, Azure Event Hubs, Azure Functions, and Logic Apps as endpoints. With this capability, individual changes to files and directories in Azure Data Lake Storage Gen2 can automatically be captured and made available to data engineers for creating rich big data analytics platforms that use event-driven architectures.

The diagram above shows a reference architecture for the modern data warehouse pipeline built on Azure Data Lake Storage Gen2 and Azure serverless components. Data from various sources lands in Azure Data Lake Storage Gen2 via Azure Data Factory and other data movement tools. Azure Data Lake Storage Gen2 generates events for new file creation, updates, renames, or deletes which are routed via Event Grid and Azure Function to Azure Databricks. A databricks job processes the file and writes the output back to Azure Data Lake Storage Gen2. When this happens, Azure Data Lake Storage Gen2 publishes a notification to Event Grid which invokes an Azure Function to copy data to Azure SQL Data Warehouse. Data is finally served via Azure Analysis Services and PowerBI.

The events that will be made available for Azure Data Lake Storage Gen2 are BlobCreated, BlobDeleted, BlobRenamed, DirectoryCreated, DirectoryDeleted, and DirectoryRenamed. Details on these events can be found in the documentation “Azure Event Grid event schema for Blob storage.”

Some key benefits include:

Seamless integration to automate workflows enables customers to build an event-driven pipeline in minutes.
Enable alerting with rapid reaction to creation, deletion, and renaming of files and directories. A myriad of scenarios would benefit from this – especially those associated with data governance and auditing. For example, alert and notify of all changes to high business impact data, set up email notifications for unexpected file deletions, as well as detect and act upon suspicious activity from an account.
Eliminate the complexity and expense of polling services and integrate events coming from your data lake with third-party applications using webhooks such as billing and ticketing systems.

Next steps

Azure Data Lake Storage Gen2 Integration with Azure Event Grid is now available in West Central US and West US 2. Subscribing to Azure Data Lake Storage Gen2 events works the same as it does for Azure Storage accounts. To learn more, see the documentation “Reacting to Blob storage events.” We would love to hear more about your experiences with the preview and get your feedback at ADLSGen2QA@microsoft.com.
Quelle: Azure

Announcing the general availability of Azure premium files

Highly performant, fully managed file service in the cloud!

Today, we are excited to announce the general availability of Azure premium files for customers optimizing their cloud-based file shares on Azure. Premium files offers a higher level of performance built on solid-state drives (SSD) for fully managed file services in Azure.

Premium tier is optimized to deliver consistent performance for IO-intensive workloads that require high-throughput and low latency. Premium file shares store data on the latest SSDs, making them suitable for a wide variety of workloads like databases, persistent volumes for containers, home directories, content and collaboration repositories, media and analytics, high variable and batch workloads, and enterprise applications that are performance sensitive. Our existing standard tier continues to provide reliable performance at a low cost for workloads less sensitive to performance variability, and is well-suited for general purpose file storage, development/test, backups, and applications that do not require low latency.

Through our initial introduction and preview journey, we’ve heard from hundreds of our customers from different industries about their unique experiences. They’ve shared their learnings and success stories with us and have helped make premium file shares even better.

“Working with clients that have large amounts of data that is under FDA or HIPAA regulations, we always struggled in locating a good cloud storage solution that provided SMB access and high bandwidth… until Azure Files premium tier. When it comes to a secure cloud-based storage that offers high upload and download speeds for cloud and on-premises VM clients, Azure premium files definitely stands out.”

– Christian Manasseh, Chief Executive Officer, Mobius Logic

“The speeds are excellent. The I/O intensive actuarial CloudMaster software tasks ran more than 10 times faster in the Azure Batch solution using Azure Files premium tier. Our application has been run by our clients using 1000’s of cores and the Azure premium files has greatly decreased our run times.”

– Scott Bright, Manager Client Data Services, PolySystems

Below are the key benefits of the premium tier. If you’re looking for more technical details, read the previous blog post “Premium files redefine limits for Azure Files.”

Performant, dynamic, and flexible

With premium tier, performance is what you define. Premium file shares’ performance can instantly scale up and down to fit your workload performance characteristics. Premium file shares can massively scale up to 100 TiB capacity and 100K IOPS with a target total throughput of 10 GiB/s. Not only do premium shares include the ability to dynamically tune performance, but also offer bursting capability to meet highly variable workload requirements with short peak periods of intense IOPS.

"We recently migrated our retail POS microservices to Azure Kubernetes Service with premium files. Our experience has been simply amazing – premium files permitted us to securely deploy our 1.2K performant Firebird databases. No problem with size or performance, just adapt the size of the premium file share to instantly scale. It improved our business agility, much needed to serve our rapidly growing customer base across multiple retail chains in France."

– Arnaud Le Roy, Chief Technology Officer, Menlog

We partnered with our internal Azure SQL and Microsoft Power BI teams to build solutions on premium files. As a result, Azure Database for PostgreSQL and Azure Database for MySQL recently opened a preview of increased scale of 16 TiB databases with 20,000 IOPS powered by premium files. Microsoft Power BI announced a powerful 20 times faster enhanced dataflows compute engine preview built upon Azure Files premium tier.

Global availability with predictable cost

Azure Files premium tier is currently available in 19 Azure regions globally. We are continually expanding regional coverage. You can check the Azure region availability page for the latest information.

Premium tier provides the most cost-effective way to create highly-performant and highly-available file shares in Azure. Pricing is simple and cost is predictable–you only pay a single price per provisioned GiB. Refer to the pricing page for additional details.

Seamless Azure experience

Customers receive all features of Azure Files in this new offering, including snapshot/restore, Azure Kubernetes Service and Azure Backup integration, monitoring, hybrid support via Azure File Sync, Azure portal, PowerShell/CLI/Cloud Shell, AzCopy, Azure Storage Explorer support, and the list goes on. Developers can leverage their existing code and skills to migrate applications using familiar Azure Storage client libraries or Azure Files REST APIs. The opportunities for future integration are limitless. Reach out to us if you would like to see more.

With the availability of premium tier, we’re also enhancing the standard tier. To learn more, visit the onboarding instructions for the standard files 100 TiB preview.

Get started and share your experiences

It is simple and takes two minutes to get started with premium file shares. Please see detailed steps for how to create a premium file share.

Visit Azure Files premium tier documentation to learn more. As always, you can share your feedback and experiences on the Azure Storage forum or email us at azurefiles@microsoft.com. Post your ideas and suggestions about Azure Storage on our feedback forum.
Quelle: Azure