Visual data ops for Apache Kafka on Azure HDInsight, powered by Lenses

This blog was written in collaboration with Andrew Stevenson, CTO at Lenses.

Apache Kafka is one of the most popular open source streaming platforms today. However, deploying and running Kafka remains a challenge for most. Azure HDInsight addresses this challenge by providing:

Ease-of-use: Quickly deploy Kafka clusters in the cloud and integrate simply with other Azure services.
Higher scale and lower total-cost-of-operations (TCO): With managed disks, compute and storage are separated, enabling you to have 100s of TBs on a cluster.
Enhanced security: Bring your own key (BYOK) encryption, custom virtual networks, and topic level security with Apache Ranger.

But that’s not all – you can now successfully manage your streaming data operations, from visibility to monitoring, with Lenses, an overlay platform now generally available as part of the Azure HDInsight application ecosystem, right from within the Azure portal!

With Lenses, customers can now:

Easily look inside Kafka topics
Inspect and modify streaming data using SQL
Visualize application landscapes

Look inside Kafka topics

A typical production Kafka cluster has thousands of topics. Imagine you want to get a high level view on all of these topics. You may want to understand the configuration of the various topics, such as the replication or partition distribution. Or you may want to look deeper inside a specific topic, investigating the message throughput and the leader broker.

While many of these insights can be provided through the Kafka CLI, Lenses greatly simplifies the experience by unifying key insights for topics and brokers via a simple to use and intuitive visual interface. With Lenses, inspecting your Kafka cluster is effortless.

Inspect and modify streaming data using SQL

What if you want to inspect the data within the Kafka topic and view the messages sent within a certain time frame? Or if you actually want to process a subset of that stream and write it back to another Kafka topic. You can achieve that with SQL queries and Processors within the Lenses UI. You can write SQL queries to validate your streaming data and unblock your client organizations faster.

SQL Processors can be deployed and monitored to perform real-time transforms and analytics, supporting all the features you would expect in SQL, like joins and aggregations. You can also configure Lenses to scale out processing with Azure Kubernetes Service (AKS).

Visualize application landscapes

At the end of the day, you’re trying to create a solution that will create business impact. That solution will be composed of various microservices, data producers, and analytical engines. Lenses gives you easy insights into your application landscape, describing the running processes and the lineage of your data platform.

In the Topology view, running applications are dynamically added, recovered at startup, and the topics are included. For creating end-to-end solutions, Lenses also provides an easy way to deploy connectors from the open source Stream Reactor project, containing a large collection of Kafka Connect Connectors.

Check out the following resources to get started with Lenses on Azure HDInsight:

Create an HDInsight Kafka cluster
Lenses on Azure HDInsight

Quelle: Azure

Simplify the management of application configurations with Azure App Configuration

We’re excited to announce the public preview of Azure App Configuration, a new service aimed at simplifying the management of application configuration and feature flighting for developers and IT. App Configuration provides a centralized place in Microsoft Azure for users to store all their application settings and feature flags (a.k.a., feature toggles), control their accesses and deliver the configuration data where it is needed.

Eliminate hard-to-troubleshoot errors across distributed applications

Companies throughout industries are transforming into digital organizations in order to better serve their customers, foster tighter relationships and respond to competition faster. We have witnessed a rapid growth in the numbers of applications our customers have. Modern applications, particularly those running in a cloud, are typically made up of multiple components and distributed in nature. Spreading configuration data across these components often leads to hard-to-troubleshoot errors in production. When a company has a large portfolio of applications, these problems multiply very quickly.

With App Configuration, you can keep your application settings together so that:

You have a single consolidated view of all configuration data.
You can easily make changes to settings, compare values, and perform rollbacks.
You have numerous options to deliver these settings to your application, including injecting them directly into your compute service (e.g., App Service), embedding in a CI/CD pipeline, or retrieving them on-demand inside your code.

App Configuration allows you to maintain control over the configuration data and handle it with confidence.

Increase release velocity with feature flags

One of the core solutions we provide with App Configuration is feature management. Traditionally, a new application feature needs to go through a series of testing before it can be released. This generally leads to long development cycles. Newer software engineering methodologies, such as feature management using feature flags, help shorten the cycles by enabling real test in production while safeguarding the application stability. Feature management solves a multitude of developer challenges especially when building applications for the cloud.

App Configuration provides built-in support for feature management. You can leverage it to remotely control feature availability in your deployed application. While it can be used with any programming language, through its REST APIs, the .NET Core and ASP.NET Core libraries offer a complete end-to-end solution out of the box.

Get started now

App Configuration provides a complete turnkey solution for dealing with application settings and feature flags. It’s easy to onboard and use. You can find the complete documentation at, “Azure App Configuration Preview documentation.” Please give it a try and let us know what you think!
Quelle: Azure

Isolate app integrations for stability, scalability, and speed with an integration service environment

Innovation at scale is a common challenge facing large organizations. A key contributor to the challenge is the complexity in coordinating the sheer number of apps and environments.

Integration tools, such as Azure Logic Apps, give you the flexibility to scale and innovate as fast as you want, on-premises or in the cloud. This is a key capability you need to have in place when migrating to the cloud, or even if you're cloud native. Often, integration has been relegated as something to do after the fact. In the modern enterprise, however, application integration is something that has to be done in conjunction with application development and innovation.

An integration service environment is the ideal solution for organizations concerned about noisy neighbor issues, data isolation, or who need more flexibility and configurability than the core Logic Apps service offers.

Building upon the existing set of capabilities, we are releasing a number of new, exciting changes that make integration service environments even better, such as:

Faster deployment times by halving the previous provisioning time

Higher throughput limits for an individual Logic App and connectors

An individual Logic App can now run for up to a year (365 days)

Integration service environment for Logic Apps is the next step for organizations who are pursuing integration as part of their core application development strategy. Here’s what an integration service environment can offer:

Direct, secure access to your virtual network resources. Enables Logic Apps to have secure, direct access to private resources, such as virtual machines, servers, and other services in your virtual network including Azure services with service endpoints and on-premises resources via Azure ExpressRoute or site to site VPN.

Consistent, highly reliable performance. Eliminates the noisy neighbor issue, removing fear of intermittent slowdowns that can impact business critical processes with a dedicated runtime where only your Logic Apps execute in.

Isolated, private storage. Sensitive data subject to regulation is kept private and secure, opening new integration opportunities.

Predicable pricing. Provides a fixed monthly cost for Logic Apps. Each integration service environment includes the free usage of one standard integration account and one enterprise connector. If your Logic Apps action execution count exceeds 50 million action executions per month, the integration service environment could provide better value.

New to integration service environments for Logic Apps? Watch this Azure Friday introduction video for assistance.

Get started with an integration service environment for Azure Logic Apps today.
Quelle: Azure

Key causes of performance differences between SQL managed instance and SQL Server

Migrating to a Microsoft Azure SQL Database managed instance provides a host of operational and financial benefits you can only get from a fully managed and intelligent cloud database service. Some of these benefits come from features that optimize or improve overall database performance. After migration many of our customers are eager to compare workload performance with what they experienced with on-premises SQL Server, and sometimes they're surprised by the results. In many cases, you might get better results on the on-premises SQL Server database because a SQL Database managed instance introduces some overhead for manageability and high availability. In other cases, you might get better results on a SQL Database managed instance because the latest version of the database engine has improved query processing and optimization features compared to older versions of SQL Server.

This article will help you understand the underlying factors that can cause performance differences and the steps you can take to make fair comparisons between SQL Server and SQL Database.

If you're surprised by the comparison results, it's important to understand what factors could influence your workload and how to configure your test environments to ensure you have a fair comparison. Some of the top reasons why you might experience lower performance on a SQL Database managed instance compared to SQL Server are listed below. You can mitigate some of these by increasing and pre-allocating file sizes or adding cores; however, the others are prerequisites for guaranteed high availability and are part of the PaaS service.

Simple or bulk recovery model

The databases placed on the SQL Database managed instance are using a full database recovery model to provide high availability and guarantee no data loss. In this scenario, one of the most common reasons why you might get worse performance on a SQL Database managed instance is the fact that your source database uses a simple or bulk recovery model. The drawback of the full recovery model is that it generates more log data than the simple/bulk logged recovery model, meaning your DML transaction processing in the full recovery model will be slower.

You can use the following query to determine what recovery model is used on your databases:

select name, recovery_model_desc from sys.databases

If you want to compare the workload running on SQL Server and SQL Database managed instances, for a fair comparison make sure the databases on both sides are using the full recovery model.

Resource governance and HA configuration

SQL Database managed instance has built-in resource governance that ensures 99.99% availability, and guarantees that management operations such as automated backups will be completed even under high workloads. If you don’t use similar constraints on your SQL Server, the built-in resource governance on SQL Database managed instance might limit your workload.

For example, there's an instance log throughput limit (up to 22MBs on the general purpose and up to 48MBs on the business critical tier) that ensures you can't load more data than the instance can backup. In this case, you might see higher INSTANCE_LOG_GOVERNOR wait statistics that don’t exist in your SQL Server instance. These resource governance constraints might slow down operations such as bulk load or index rebuild because these operations require higher log rates.

In addition, the secondary replicas in business critical tier instances might slow down the primary database if they can't catch-up the changes and apply them, so you might see additional HADR_DATABASE_FLOW_CONTROL or HADR_THROTTLE_LOG_RATE_SEND_RECV wait statistics.

If you're comparing your SQL Server workload running on local SSD storage to the business critical tier, note that the business critical instance is an Always On availability group cluster with three secondary replicas. Make sure that your source SQL Server has an HA implementation similarly using Always On availability groups with at least one synchronous commit replica. If you're comparing the business critical tier with a single SQL Server instance writing to the local disk, this would be an unrealistic comparison due to the absence of HA on your source instance. If you are using async always on replicas, you would have HA with better performance, but in this case you are making the trade-off between the possibility of data-loss in favor of performance, and you will get the better results on the SQL Server instance.

Automated backup schedule

One of the main reasons why you would choose the SQL Database managed instance is the fact that it guarantees you will always have backups of your databases, even under heavy workloads. The databases in a SQL Database managed instance have scheduled full, incremental, and log backups. Full backups are taken every seven days, incremental every twelve hours, and log backups are taken every five to ten minutes. If you have multiple databases on the instance there's a high chance there is at least one backup currently running.

Since the backup operations are using some instance resources (CPU, disk, network), they can affect workload performance. Make sure the databases on the system that you compare with the managed instance have similar backup schedules. Otherwise, you might need to accept that you're getting better results on your SQL Server instance because you're making a trade-off between database recovery and performance, which is not possible on a SQL Database managed instance.

If you're seeing unexpected performance differences, check if there is some ongoing full/differential backup either on the SQL Database managed instance or SQL Server instance that can affect performance of the currently running workload, using the following query:

SELECT r.command, query = a.text, start_time, percent_complete,
eta = dateadd(second,estimated_completion_time/1000, getdate())
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) a
WHERE r.command IN ('BACKUP DATABASE','BACKUP LOG')

If you see currently running full or incremental backup during the short-running benchmark, you might pause your workload and resume it once the backup finishes.

Connection and App to Database proximity

The application accessing the databases and executing the benchmark queries on the SQL Database managed instance and SQL Server instance must be in a similar network proximity range in both cases. If you are placing your application and SQL Server database in the local environment (or running an app like HammerDB from the same machine where the SQL Server is installed) you will get better results on SQL Server compared to the SQL Database managed instance, which is placed on a distributed cloud environment with respect to the application. Make sure that in both cases you're running the benchmark application or query on separate virtual machines in the same region as SQL Database managed instance to get the valid results. If you're comparing an on-premises environment with the equivalent cloud environments, try to measure bandwidth and latency between the app and database and try to ensure they are similar.

SQL Database managed instance is accessed via proxy gateway nodes that accept the client requests and redirect them to the actual database engine nodes. In order to provide the results closer to your environment, enable ProxyOverride mode on your instance using Set-AzSqlInstance PowerShell command to enable direct access from the client to the nodes currently hosting your SQL Database managed instance.

In addition, due to compliance requirements, a SQL Database managed instance enforces SSL/TLS transport encryption which is always enabled. Encryption can introduce overhead in case of a large number of queries. If your on-premises environment does not enforce SSL encryption you will see additional network overhead in the SQL Database managed instance.

Transparent data encryption

The databases on SQL Database managed instance are encrypted by default using Transparent Data Encryption. Transparent Data Encryption encrypts/decrypts every page that is exchanged with the disk storage. This spends more CPU resources, and introduces additional latency in the process of fetching and saving the data pages to or from disk storage. Make sure that both databases on SQL Database managed instance and SQL Server have Transparent Data Encryption either turned on or off, and that database encryption/decryption operations have completed before starting performance testing.

You can use the following query to determine whether the databases are encrypted:

select name, is_encrypted from sys.databases

Another important factor that might affect your performance is encrypted TempDB. TempDB is encrypted if at least one database on your SQL Server or SQL Database managed instance is encrypted. As a result, you might compare two databases that are not encrypted, but due to some other SQL Database managed instance being encrypted (although it's not involved in the workload) the TempDB will also be encrypted. The unencrypted databases will still use encrypted TempDB and any query that creates temporary objects or uses spills would be slower. Note that TempDB will only get decrypted once all user databases on an instance are decrypted, and the instance restarts. Scaling a SQL Database managed instance to a new pricing tier and back is one way to restart it.

Database engine settings

Make sure the database engine setting such as database compatibility levels, trace flags, system configurations (‘cost threshold for parallelism’, ’max degree of parallelism’), database scoped configurations (LEGACY_CARDINALITY_ESTIMATOR, PARAMETER_SNIFFING, QUERY_OPTIMIZER_HOTFIXES, etc.), and database settings (AUTO_UPDATE_STATISTICS, DELAYED_DURABILITY) on the SQL Server and SQL Database managed instances are the same on both databases.

The following sample queries can help you to identify setting on SQL Server and Azure SQL Database managed instance:

select compatibility_level, snapshot_isolation_state_desc, is_read_committed_snapshot_on,

  is_auto_update_stats_on, is_auto_update_stats_async_on, delayed_durability_desc
from sys.databases;
GO

select * from sys.database_scoped_configurations;
GO

dbcc tracestatus;
GO

select * from sys.configurations;

Compare the results of these queries on the SQL Database managed instance and SQL Server and try to align the differences if you identify some.

Note: The list of trace flags and configurations might be very long so we recommend filtering them or lookng only on the trace flags you've changed or know are affecting performance. Some of the trace flags are pre-configured on SQL Database managed instance as part of PaaS configurations and they are not affecting performance.

You might experiment with changing the compatibility level to a higher value, turning on the legacy cardinality estimator, or the automatic tuning feature on the SQL Database managed instance, which might give you better results than your SQL Server database.

Also note that SQL Database managed instance might provide better performance even if you align all parameters because it has the latest improvements, or fixes that are not bound to compatibility level, or some features, like forcing last good plan, that might improve your workload.

Hardware and environment specification

SQL Database managed instance runs on standardized hardware with pre-defined technical characteristics that are probably different than your environment. Some of the characteristics you might need to consider when comparing your environment with the environment where the SQL Database managed instance is running are:

Number of cores should be the same both on SQL Server and the SQL Database managed instance. Note that a SQL Database managed instance uses 2.3-2.4 GHz processors, which might be different than your processor speed. It might consume more or less CPU for the same operation due to the CPU differences. If possible, check if hyperthreading is used on the SQL Server environment when comparing to the Gen4 and Gen5 hardware generations on a SQL Database managed instance. One on Gen4 hardware does not use hyperthreading, while on Gen5 it does. If you are comparing SQL Server running on a bare-metal machine with a SQL Database managed instance or SQL Server running on a virtual machine you'll probably get better results on a bare-metal instance.
Amount of memory including memory/core ratio (5.1 GB/core on Gen5, 7 GB/core on Gen4). Higher memory/core ratio provides bigger buffer pool cache and increases cache hit ratio. If your workload does not perform well on a managed interface with the memory/core ratio 5, then you probably need to choose a virtual machine with the appropriate memory/core ratio instead of a SQL Database managed instance.
IO characteristics – You need to be aware that performance of the storage system might be very different compared to your on-premises environment. A SQL Database managed instance is a cloud database and relies on Azure cloud infrastructure.

The general purpose tier uses remote Azure Premium disks where IO performance depends on the file sizes. If you reach the log limit that depends on the file size, you might notice WRITE_LOG waits and less IOPS in file statistics. This issue might occur on a SQL Database managed instance if the log files are small and not pre-allocated. You might need to increase the size of some files in the general purpose tier to get better performance (see this Tech Community article Storage performance best practices and considerations for Azure SQL Managed Instance General Purpose tier).
A SQL Database managed instance does not use instant file initialization, so you might see additional PREEMPTIVE_OS_WRITEFILEGATHER wait statistics since the date files are filled with zero bytes during file growth.

Local or remote storage types – Make sure you're considering local SSD versus remote storage while doing the comparison. The general purpose tier uses remote storage (Azure Premium Storage) that can't match your on-premises environment if it uses local SSD or a high-performance SAN. In this case you would need to use the business critical tier as a target. The general purpose tier can be compared with other cloud databases like SQL Server on Azure Virtual Machines that also use remote storage (Azure Premium Storage). In addition, beware that remote storage used by a general purpose instance is still different than remote storage used by a SQL Virtual Machine because:

The general purpose tier uses a dedicated IO resource per each database file that depends on the size of the individual files, while SQL Server on Azure Virtual Machine uses shared IO resources for all files where IO characteristics depend on the size of the disk. If you have many small files, you will get better performance on a SQL Virtual Machine, while you can get better performance on a SQL Database managed instance if the usage of files can be parallelized because there are no noisy neighbors who are sharing the same IO resources.
SQL Virtual Machines use a read-caching mechanism that improves read speed.

If your hardware specs and resource allocation are different, you might expect different performance results that can be resolved only by changing the service tier or increasing file size. If you are comparing a SQL Database managed instance with SQL Server on Azure Virtual Machines, make sure that you are choosing a virtual machine series that has memory/cpu ratio similar to SQL Database managed instance, such as DS series.

Azure SQL Database managed instance provides a powerful set of tools that can help you troubleshoot and improve performance of your databases, in addition to built-in intelligence that could automatically resolve potential issues. Learn more about monitoring and tuning capabilities of Azure SQL Database managed instance in the following article: https://docs.microsoft.com/en-us/azure/sql-database/sql-database-monitoring-tuning-index
Quelle: Azure

Simplifying event-driven architectures with the latest updates to Event Grid

Event-driven architectures are increasingly replacing and outpacing less dynamic polling-based systems, bringing the benefits of serverless computing to IoT scenarios, data processing tasks or infrastructure automation jobs. As the natural evolution of microservices, companies all over the world are taking an event-driven approach to create new experiences in existing applications or bring those applications to the cloud, building more powerful and complex scenarios every day.

Today, we’re incredibly excited to announce a series of updates to Event Grid that will power higher performance and more advanced event-driven applications in the cloud:

Public preview of IoT Hub device telemetry events
Public preview of Service Bus as an event handler
Automatic server-side geo-disaster recovery
General availability of Event Domains, now with up to 100K topics per Domain
Public preview of 1MB event support
List search and pagination APIs
General availability of advanced filters with increased depth of filtering

Expanded integration with the Azure ecosystem

One of the biggest features we have been asked for since launching the Azure IoT Hub integration with Event Grid is device telemetry events. Today, we’re finally enabling that feature in public preview in all public regions except East US, West US, and West Europe. We are excited for you to try this capability and build more streamlined IoT solutions for your business.

Subscribing to device telemetry events allows you to easily integrate data from your devices into your solution more easily, including serverless applications using Azure Functions or Azure Logic Apps, and any other services by using webhooks, whether they are on Azure or not. This helps simplify IoT architectures by eliminating the need of additional services that poll for device telemetry for further processing.

By publishing device telemetry events to Event Grid, IoT Hub expands the services your data can reach, beyond the endpoints supported through message routing. For example, you can automate downstream workflows by creating different subscriptions to device telemetry events for different device types, identified by the device twin tag, and triggering distinct Azure Functions or third party applications for unique computation per device type. Based on your Event Grid subscriptions to device telemetry events, we create a default route in IoT hub, handling all your Event Grid subscriptions to device telemetry.

Learn more about IoT Hub device telemetry in docs, and continue to submit your suggestions through the Azure IoT User Voice forum.

We are also adding Service Bus as an event handler for Event Grid in public preview, so starting today you can route your events in Event Grid directly to Service Bus queues. Service Bus can now act as either an event source or event handler, making for a more robust experience delivering events and messages in distributed enterprise applications. It is currently in public preview and does not work with Service Bus topics and sessions, but it does work with all tiers of Service Bus queues.

This enables command and control scenarios in which you receive events of activity on other services such as blob created, device created, and job finished passing them along for further processing.

Learn more about Service Bus as a destination in docs.

Server-side geo disaster recovery

Event Grid now has built-in automatic geo disaster recovery (GeoDR) of metadata, applicable to all existing Domains, Topics and Event Subscriptions, not just for new ones. This provides a vastly improved resilience against service interruptions, all fully managed by our platform. In the event of an outage that takes out an entire Azure region, the Event Grid service will already have all of your eventing infrastructure metadata synced to a paired region, and your new events will begin to flow again with no intervention from your side required, avoiding service interruption automatically.

Disaster recovery is generally measured with two metrics:

Recovery Point Objective (RPO): the minutes or hours of data that may be lost.
Recovery Time Objective (RTO): the minutes of hours the service may be down.

Event Grid’s automatic failover has different RPO’s and RTO’s for your metadata (event subscriptions, plus more) and data (events). If you need different specification from below, you can still always implement your own client-side failover using the topic health APIs.

Metadata RPO: Zero minutes. You read that right. Any time a resource is created in Event Grid, its instantly replicated across regions. In the event of a failover, no metadata is lost.
Metadata RTO: Though generally this happens much more quickly, within 60 minutes Event Grid will begin to accept create/update/delete calls for topics and subscriptions.
Data RPO: If your system is healthy and caught up on existing traffic at the time of regional failover, the RPO for events is about 5 minutes.
Data RTO: Like metadata, this generally happens much more quickly, however within 60 minutes Event Grid will begin accepting new traffic after a regional failover.

Here’s the best part, there is no cost for metadata GeoDR on Event Grid. It is included on the current price of the service and won’t incur any additional charges.

Powering advanced event-driven workloads

As we see more advanced event-driven architectures for diverse scenarios such as IoT, CRM, or financial we’ve noticed an increasing need on expanding our capabilities for multitenant applications and workloads handling bigger amount of data in their events.

Event Domains give you the power to organize your entire eventing infrastructure under a single construct, set fine grain auth rules on each topic for who can subscribe, and manage all event publishing with a single endpoint. Classic pub-sub architectures are built exclusively on topics and subscriptions, however as you build more advanced and hi-fidelity event-driven architectures, the burden on maintenance increases exponentially. Event Domains take the headache out of it by handling much of the management for you.

Today we’re happy to announce that Event Domains are now generally available, and with that, you’ll be able to have 100,000 topics per Domain. Here’s the full set of Event Domains limits with general availability:

100,000 topics per Event Domain
100 Event Domains per Azure Subscription
500 event subscriptions per topic in an Event Domain
50 ‘firehose’ event subscriptions at the Event Domain scope
5,000 events/second into an Event Domain

As always, if these limits don’t suit you, feel free to reach out via support ticket or by emailing askgrid@microsoft.com

so we can get you higher capacity.

We also acknowledge that advanced event-driven architectures don’t always fit in the confines of 64 KB. These workloads require handling larger events for a simpler architecture, and today we’re announcing the public preview of events up to 1MB.

There are no configuration changes required and this will work on existing event subscriptions, and everything under 64 KB will be still be covered by our general availability SLA. To try it out, just push larger events, noticing that events over 64 KB will be charged in 64 KB increments, and the batch size limit for events sent to Event Grid as a JSON array is still 1MB in total.

Simplified management of events

You might have thousands of event subscriptions or, with the general availability of Event Domains, hundreds of thousands of topics floating around your Azure subscription. In order to make searching and managing of these resources easier, we’ve introduced list search and list pagination APIs throughout Event Grid. For more information check out all the details in our, “Azure Event Grid Documentation.”

Advanced filters used to route messages in Event Grid are now generally available, with no restriction on the number of nested objects in your JSON. This allows for more granularity when filtering events before passing it to other services for further processing, reducing compute time, and resources needed by avoiding performing this filtering elsewhere.

If you haven’t played with advanced filters yet, you can use the following operators on any part of the event, making the possibilities nearly endless: StringContains, StringBeginsWith, StringEndsWith, StringIn, StringNotIn, NumberGreaterThan, NumberGreaterThanOrEquals, NumberLessThan, NumberLessThanOrEquals, NumberIn, NumberNotIn, BoolEquals.

Get started today

As always, we love to hear your thoughts, feedback, and wish lists as you get a chance to try out these new features! You can start now with the following resources, and please reach out with your feedback.

Sign up for an Azure free account if you don’t have one yet
Subscribe to IoT Hub device telemetry events with Event Grid
Learn more about using Service Bus as an event handler
Build more powerful multitenant applications with Event Domains
Perform searches and pagination over thousands and thousands of events with these new APIs
Route only the necessary events for processing using advanced filters

Quelle: Azure

Manage your cross cloud spend using Azure Cost Management

It’s common for enterprises to run workloads on more than one cloud provider; however, adopting a multi-cloud strategy comes with complexities like handling different cost models, varying billing cycles, and different cloud designs that can be difficult to navigate across multiple dashboards and views.

We’ve heard from many of you that you need a central cost management solution built to help you manage your spend across multiple cloud providers, prevent budget overruns, maintain control, and create accountability with your consumers.

Azure Cost Management now offers cross-cloud support. This is available in preview and can play a critical role in helping you efficiently and effectively managing your organization’s multi-cloud needs. Onboard your AWS costs via an easy-to-use cloud connector to get a single view of your expenditures, so you can analyze and budget for the future in one place.

Get started now and create your first AWS connector with the free preview to explore these capabilities and more.

For more information about managing AWS costs, see this Manage AWS costs and usage in Azure documentation.

What’s available in this preview:

Analyzing Azure and AWS spend with cost analysis

Get deep insights into your cloud spend with the rich graphic capabilities in cost analysis. Analyze your costs across 18 available dimensions like provider, service name, usage location, availability zones, meter, and tags. Cost analysis provides the flexibility to view granular charges to understand what’s driving costs and anomalies, or high-level views to analyze total and month-over-month costs for one or both of your cloud providers. Learn more about Azure Cost Management cost analysis.

 

Budgets and alerts across your Azure & AWS spend

Budgets and alerts in Cost Management help you plan for and drive organizational accountability by allowing you to set spend amount limits and alert thresholds. You can now set budgets and receive alerts on AWS scopes, as well. You can set budgets for various scopes like subscriptions, resource groups, or for your enterprise agreement hierarchy if you are an enterprise customer. Budget notifications are integrated with Azure Action Groups, which enables email, SMS, or you can plug into your automation scripts using available Webhook and Functions integration. Learn more about Azure Cost Management budgets.

 

What’s coming next?

With Azure Cost Management, Microsoft is committed to supporting you manage your multi-cloud environment. We will continue to develop additional Cost Management features so you can benefit from a unified user experience across both Azure and AWS.
In the coming months we’ll be focusing on building features like the ability to save and schedule reports, additional capabilities in cost analysis, budgets, forecast, alerts, exports, and show-backs. And looking further into the future, we plan to continue enhancing our multi-cloud support for additional cloud products.
Follow us at @AzureCostMgmt on Twitter for more updates and information.

Quelle: Azure

Azure NetApp Files is now generally available

Today, we're excited to announce the General Availability (GA) of Azure NetApp Files, the industry’s first bare-metal cloud file storage and data management service. Azure NetApp Files is an Azure first-party service for migrating and running the most demanding enterprise file-workloads in the cloud including databases, SAP, and high-performance computing applications with no code changes. Today’s milestone is the result of deep investment by both companies to provide a great experience for our customers through a service that’s unique in the industry.

Since launching in preview mode, several Fortune 100 enterprises across the world have provided valuable feedback that helped us enrich the service.

"We wanted an on-premise like performance for our reservoir simulation and analysis software. We were thrilled to see Azure NetApp Files exceeding our expectations with an over 5x performance increase. Most importantly, the massive scale-up/down capability of Azure NetApp Files now allows for pure cloud-based consumption of both capacity AND performance,” said Juan Pedro Brett, Digital Transformation Engineer, E&P at Repsol.

Azure NetApp Files represents the culmination of a deep partnership between Microsoft and NetApp, combining NetApp’s proven and industry-leading ONTAP technology with the scale, reach, and enterprise capabilities of Azure.

“Azure NetApp Files is the most significant proof-point of NetApp’s commitment to accelerate our customer’s digital transformation, said Anthony Lye, Senior Vice President and General Manager of Cloud BU at NetApp. “We have partnered extensively with Microsoft across engineering, sales, and marketing to help customers benefit from the availability of our industry leading storage and data management technology on Azure. The general availability of Azure NetApp Files is also a fantastic moment to thank our customers for their enormous interest and feedback that has helped shaped this service.”

Read more about NetApp’s perspective on this announcement on the NetApp blog.

Deep integration of NetApp’s industry leading ONTAP storage and data management technology into Azure provides unique value to our customers in several ways:

Seamless Azure experience

Azure NetApp Files is a fully managed cloud service with full Azure portal integration and access via REST API and Azure SDKs, and soon via Azure CLI and PowerShell. It’s sold and supported exclusively by Microsoft. Customers can seamlessly migrate and run applications in the cloud without worrying about procuring or managing storage infrastructure. Additionally, customers can purchase Azure NetApp Files and get support through existing Azure agreements, with no up-front or separate term agreement.

Power of NetApp ONTAP

NetApp’s ONTAP systems serve hundreds of thousands of customers and has earned the trust of enterprise organizations over decades. The technology provides proven protocol support, including support for NFSv3 and SMB 3.1. It enables powerful data management with snapshots of datasets, high availability, and can achieve sub-millisecond latency.

Advanced security for business-critical data

Azure NetApp Files is built and operated with Azure’s industry-leading standards and processes for security, and benefits from the multi-layered security provided by Microsoft across its physical datacenters, infrastructure, and operations. Azure NetApp Files provides FIPS-140-2-compliant data encryption at rest, role-based access control (RBAC), Active Directory authentication (enabled for SMB), and export policies for network-based access control.

Support for hybrid scenarios

Azure NetApp Files enables easy migration of data across on-premises and cloud infrastructures using Cloud Sync, a NetApp service for rapid, security-enhanced data synchronization. This simplifies lift and shift and DevOps scenarios with capabilities like instantaneous snapshot, restore, and Active Directory integration (for SMB) that work the same way in the cloud and on premises. Integrated data replication and backup will be available in the near future. Learn more about Cloud Sync.

Get started with Azure NetApp Files

You can request onboarding to Azure NetApp Files by submitting an online request or by reaching out to your Microsoft representative.

We will continue to strongly partner with NetApp and look forward to hearing your feedback on Azure NetApp Files. You can email us at ANFFeedback@microsoft.com or share your ideas and suggestions for Azure Storage on our feedback forum.
Quelle: Azure

Azure.Source – Volume 84

Now available

All U.S. Azure regions now approved for FedRAMP High impact level

We’re now sharing our ability to provide Azure public services that meet U.S. Federal Risk and Authorization Management Program (FedRAMP) High impact level and extend FedRAMP High Provisional Authorization to Operate (P-ATO) to all of our Azure public regions in the United States. Achieving FedRAMP High means that both Azure public and Azure Government data centers and services meet the demanding requirements of FedRAMP High, making it easier for more federal agencies to benefit from the cost savings and rigorous security of the Microsoft Commercial Cloud.

Now in preview

Drive higher utilization of Azure HDInsight clusters with autoscale

We are excited to share the preview of the Autoscale feature for Azure HDInsight. This feature enables enterprises to become more productive and cost-efficient by automatically scaling clusters up or down based on the load or a customized schedule.

Announcing the preview of Windows Server containers support in Azure Kubernetes Service

Kubernetes is taking the app development world by storm. Earlier this month, we shared that the Azure Kubernetes Service (AKS) was the fastest growing compute service in Azure’s history. Today, we’re excited to announce the preview of Windows Server containers in Azure Kubernetes Service (AKS) for the latest versions, 1.13.5 and 1.14.0.  With this, Windows Server containers can now be deployed and orchestrated in AKS enabling new paths to migrate and modernize Windows Server applications in Azure.

Optimize price-performance with compute auto-scaling in Azure SQL Database serverless

Optimizing compute resource allocation to achieve performance goals while controlling costs can be a challenging balance to strike – especially for database workloads with complex usage patterns. To help address these challenges, we are pleased to announce the preview of Azure SQL Database serverless. SQL Database serverless (preview) is a new compute tier that optimizes price-performance and simplifies performance management for databases with intermittent and unpredictable usage. Line-of-business applications, dev/test databases, content management, and e-commerce systems are just some examples across a range of applications that often fit the usage pattern ideal for SQL Database serverless.

Visual interface for Azure Machine Learning service

During Microsoft Build we announced the preview of the visual interface for Azure Machine Learning service. This new drag-and-drop workflow capability in Azure Machine Learning service simplifies the process of building, testing, and deploying machine learning models for customers who prefer a visual experience to a coding experience. This capability brings the familiarity of what we already provide in our popular Azure Machine Learning Studio with significant improvements to ease the user experience.

Technical content

Kubernetes - from the beginning, Part I, Basics, Deployment and Minikube

Kubernetes is a BIG topic. In this blog series, Chris Noring tackles the basics. Part 1 covers: Why Kubernetes and orchestration in general; talking through Minikube, simple deploy example; cluster and basic commands; deploying an app; and concepts and troubleshooting of pods and nodes.

Introduction to AzureKusto

This post is to announce the availability of AzureKusto, the R interface to Azure Data Explorer (internally codenamed “Kusto”), a fast, fully managed data analytics service from Microsoft. It is available from CRAN, or you can install the development version from GitHub.

Microsoft Azure for spoiled people

In this article, the author walks you through the easiest possible way to set up a Vue.js CLI-built web app on Azure with continuous integration via GitHub.

Azure IoT Central and MXChip Hands-on Lab

This hands-on lab covers creating an Azure IoT Central application, connecting an MXChip IoT DevKit device to your Azure IoT Central application, and setting up a device template.

Azure shows

A new way to try .NET

Learning a programming language is becoming a fundamental aspect of education across the world. We're always looking for new and interesting ways to teach programming to learners at all levels. From Microsoft Build 2019,  we had Maria Naggaga come on to show us the Try .NET project. She shows us how this simple tool will allow us to create interactive documentation, workshops, and other interesting learning experiences.

Xamarin.Forms 101: Dynamic resources

Let's take a step back in a new mini-series that I like to call Xamarin.Forms 101. In each episode we will walk through a basic building block of Xamarin.Forms to help you build awesome cross-platform iOS, Android, and Windows applications in .NET. This week we will look at how to use dynamic resources to change the value of a resource while the application is running.

Cosmos DB data in a smart contract with Logic Apps

We show an IoT use case that highlights how to leverage the power of Cosmos DB to manipulate IoT data within a Cosmos DB and use that data in smart contracts via the Ethereum Logic App connector.

ARM templates and Azure policy

Cynthia talks with Satya Vel on the latest ARM template updates including an enhanced template export experience, best practices for ARM clients, and new capabilities that are now available on ARM templates.

Industries and partners

Securing the pharmaceutical supply chain with Azure IoT

You’re responsible for overseeing the transportation of a pallet of medicine halfway around the world. Drugs will travel from your pharmaceutical company’s manufacturing outbound warehouse in central New Jersey to third-party logistics firms, distributors, pharmacies, and ultimately, patients. Each box in that pallet – no bigger than the box that holds the business cards on your desk – contains very costly medicine, the product of 10 years of research and R&D spending. But there are several big catches. Read on to see what they are and how Azure IoT helps overcome them.

How you can use IoT to power Industry 4.0 innovation

IoT is ushering in an exciting—and sometimes exasperating—time of innovation. Adoption isn’t easy, so it’s important to hold a vision of the promise of Industry 4.0 in mind as you get ready for this next wave of business. This post is the fourth in a four-part series designed to help companies maximize their ROI on IoT.
Quelle: Azure

HB-series Azure Virtual Machines achieve cloud supercomputing milestone

New HPC-targeted cloud virtual machines are first to scale to 10,000 cores

Azure Virtual Machine HB-series are the first on the public cloud to scale a MPI-based high performance computing (HPC) job to 10,000 cores. This level of scaling has long been considered the realm of only the world’s most powerful and exclusive supercomputers, but now is available to anyone using Azure.

HB-series virtual machines (VMs) are  optimized for HPC applications requiring high memory bandwidth. For this class of workload, HB-series VMs are the most performant, scalable, and price-performant ever launched on Azure or elsewhere on the public cloud.

With the AMD EPYC processors, the HB-series delivers more than 260 GBs of memory bandwidth, 128 MB L3 cache, and SR-IOV-based 100 Gbs InfiniBand. At scale, a customer can utilize up to 18,000 physical CPU cores and more than 67 terabytes of memory for a single distributed memory computational workload.

For memory-bandwidth bound workloads, the HB-series delivers something many in HPC thought may never happen. Azure-based VMs are now as or more capable as bare-metal, on-premises status quo that dominates the HPC market, and at a highly competitive price point.

World-class HPC technology

HB-series VMs feature the cloud’s first deployment of AMD EPYC 7000-series CPUs explicitly for HPC customers. AMD EPYC features 33 percent more memory bandwidth than any x86 alternative, and even more than leading POWER and ARM server platforms. In context, this 263 GBs of memory bandwidth the HB-series VM delivers 80 percent more than competing cloud offerings in the same memory per core class.

HB-series VMs expose 60 non-hyperthreaded CPU cores and 240 GB of RAM, with a baseclock of 2.0 GHz, and an all-cores boost speed of 2.55 GHz. HB VMs also feature a 700 GB local NVMe SSD, and support up to four Managed Disks including the new Azure P60/P70/P80 Premium Disks.

A flagship feature of HB-series VMs is 100 GBs InfiniBand from Mellanox. HB-series VMs expose the Mellanox ConnectX-5 dedicated back-end NIC via SR-IOV, meaning customers can use the same OFED driver stack that they’re accustomed to in a bare metal context. HB-series VMs deliver MPI latencies as low as 2.1 microseconds, with consistency, bandwidth, and message rates in line with bare-metal InfiniBand deployments.

Cloud HPC scaling achievement

As part of early acceptance testing, the Azure HPC team benchmarked many widely used HPC applications. One common class of applications are those that simulate computational fluid dynamics (CFD). To see how far HB-series VMs could scale, we selected the Le Mans 100 million cell model available to Star-CCM+ customers, with results as follows:

 

The Le Mans 100 million cell model scaled to 256 VMs across multiple configurations accounting for as many as 11,520 CPU cores. Our testing revealed that maximum scaling efficiency could be had with two MPI ranks per NUMA domain yielding a top-end scaling efficiency of 71.3 percent efficiency. For top-end performance, three MPI ranks per NUMA domain yielded the fastest overall performance. Customers can choose which metric they find most valuable based on a wide variety of factors.

Delighting HPC customers on Azure

The unique capabilities and cost-performance of HB-series VMs are a big win for scientists and engineers who depend on high-performance computing to drive their research and productivity to new heights. Organizations spanning aerospace, automotive, defense, financial services, heavy equipment, manufacturing, oil & gas, public sector academic, and government research have shared feedback on how the HB-series has increased product performance and provided new insights through detailed simulation models.

Rescale partners with Azure to provide HPC resources for computationally complex simulations and analytics. Launching today, Azure Virtual Machine HB-series VM can be consumed through Rescale’s ScaleX® as the new “Amber” compute resource.

“As the only fully managed HPC cloud service in the market, Rescale creates an elegant way to move on-premises HPC workloads to the cloud. We have been waiting with great anticipation for Microsoft to introduce cloud building blocks specifically engineered for HPC," said Adam McKenzie, CTO of Rescale. "Now, new HB-series VMs on Azure enable MPI workloads to scale to tens of thousands of cores with the kind of cost-performance that rivals on-premises supercomputers”

Available now

Azure Virtual Machine HB-series are currently available in South Central US and Western Europe, with additional regions rolling out soon.

Find out more about high performance computing (HPC) in Azure
Learn about Azure Virtual Machines

Quelle: Azure

Transforming Azure Monitor Logs for DevOps, granular access control, and improved Azure integration

Logs are critical for many scenarios in the modern digital world. They are used in tandem with metrics for observability, monitoring, troubleshooting, usage and service level analytics, auditing, security, and much more. Any plan to build an application or IT environment should include a plan for logs.

Logs architecture

There are two main paradigms for logs:

Centralized: All logs are kept in a central repository. In this scenario, it is easy to search across resources and cross-correlate logs but, since these repositories get big and include logs from all kind of sources, it's hard to maintain access control on them. Some organizations completely avoid centralized logging for that reason, while other organizations that use centralized logging restrict access to very few admins, which prevents most of their users from getting value out of the logs.

Siloed: Logs are either stored within a resource or stored centrally but segregated per resource. In these instances, the repository can be kept secure, and access control is coherent with the resource access, but it's hard or impossible to cross-correlate logs. Users who need a broad view of many resources cannot generate insights. In modern applications, problems and insights span across resources, making the siloed paradigm highly limited in its value.

To accommodate the conflicting needs of security and log correlations many organizations have implemented both paradigms in parallel, resulting in a complex, expensive, and hard-to-maintain environment with gaps in logs coverage. This leads to lower usage of log data in the organization and results in decision-making that is not based on data.

New access control options for Azure Monitor Logs

We have recently announced a new set of Azure Monitor Logs capabilities that allow customers to benefit from the advantages of both paradigms. Customers can now have their logs centralized while seamlessly integrated into Azure and its role based access control (RBAC) mechanisms. We call this resource-centric logging. It will be added to the existing Azure Monitor Logs experience automatically while maintaining the existing experiences and APIs. Delivering a new logs model is a journey, but you can start using this new experience today. We plan to enhance and complete alignment of all Azure Monitor's components over the next few months.

The basic idea behind resource-centric logs is that every log record emitted by an Azure resource is automatically associated with this resource. Logs are sent to a central workspace container that respects scoping and RBAC based on the resources. Users will have two options for accessing the data:

Workspace-centric: Query all data in a specific workspace–Azure Monitor Logs container. Workspace access permissions apply. This mode will be used by centralized teams that need access to logs regardless of the resource permissions. It can also be used for components that don't support resource-centric or off-Azure resources, though a new option for them will be available soon.

 Resource-centric: Query all logs related to a resource. Resource access permissions apply. Logs will be served from all workspaces that contain data for that resource without the need to specify them. If workspace access control allows it, there is no need to grant the users access to the workspace. This mode works for a specific resource, all resources in a specific resource group, or all resources in a specific subscription. Most application teams and DevOps will be using this mode to consume their logs.

Azure Monitor experience automatically decides on the right mode depending on the scope the user chooses. If the user selects a workspace, queries will be sent in workspace-centric mode. If the user selects a resource, resource group, or subscription, resource-centric is used. The scope is always presented in the top left section of the Log Analytics screen:

You can also query all logs of resources in a specific resource group using the resource group screen:

Soon, Azure Monitor will also be able to scope queries for an entire subscription.

To make logs more prevalent and easier to use, they are now integrated into many Azure resource experiences. When log search is opened from a resource menu, the search is automatically scoped to that resource and resource-centric queries are used. This means that if users have access to a resource, they'll be able to access their logs. Workspace owners can block or enable such access using the workspace access control mode.

Another capability we're adding is the ability to set permissions per table that store the logs. By default, if users are granted access to workspaces or resources, they can read all their log types. The new table RBAC allows admins to use Azure custom roles to define limited access for users, so they're only able to access some of the tables, or admins can block users from accessing specific tables. You can use this, for example, if you want the networking team to be able to access only the networking related table in a workspace or a subscription.

As result of these changes, organizations will have simpler models with fewer workspaces and more secure access control. Workspaces now assume the role of a manageable container, allowing administrators to better govern their environments. Users are now empowered to view logs in their natural Azure context, helping them to leverage the power of logs in their day-to-day work.

The improved Azure Monitor Logs access control lets you now enjoy both worlds at once without compromise on usability and security. Central teams can have full access to all logs while DevOps teams can access logs only for their resources. This comes on top of the powerful log analytics, integration and scalability capabilities that are used by tens of thousands of customers.

Next steps

To use it today, you need to:

Decide which workspaces should be used to store all data. Take into account billing, regulation, and data ownership.
Change your workspace access control mode to “Use resource or workspace permissions” to enable them for resource-centric access. Workspaces created after March 2019 are configured to this mode by default.
Remove workspace access permissions from your application teams and DevOps.
Let your users become master of their logs.

Quelle: Azure