New Project Flash Update: Advancing Azure Virtual Machine availability monitoring

“Earlier this year, we introduced Project Flash in the Advancing Reliability blog series, to reaffirm our commitment to empowering Azure customers in monitoring virtual machine (VM) availability in a robust and comprehensive manner. Today, we’re excited to share the progress we’ve made since then in developing holistic monitoring offerings to meet customers’ distinct needs. I’ve asked Senior Technical Program Manager, Pujitha Desiraju, from the Azure Core Production Quality Engineering team to share the latest investments as part of Project Flash, to deliver the best monitoring experience for customers.”—Mark Russinovich, CTO, Azure.

Flash, as the project is internally known, is a collection of efforts across Azure Engineering, that aims to evolve Azure’s virtual machine (VM) availability monitoring ecosystem into a centralized, holistic, and intelligible solution customers can rely on to meet their specific observability needs. As part of this multi-year endeavor, we’re excited to announce the:

General availability of VM availability information in Azure Resource Graph for efficient and at-scale monitoring, convenient for detailed downtime investigations and impact assessment.
Preview of a VM availability metric in Azure Monitor for quick debugging is now publicly available, trend analysis of VM availability over time, and setting up threshold-based alerts on scenarios that impact workload performance.
Preview of VM availability status change events via Azure Event Grid for instantaneous notifications on critical changes in VM availability, to quickly trigger remediation actions to prevent end-user impact.

Our commitment remains, to maintaining data consistency and similar rigorous quality standards across all the monitoring solutions that are part of Flash, including existing solutions like Resource Health or Activity Log, so we deliver a consistent and cohesive experience to customers.

VM availability information in Azure Resource Graph for at-scale analysis

In addition to already flowing VM availability states, we recently published VM health annotations to Azure Resource Graph (ARG) for detailed failure attribution and downtime analysis, along with enabling a 14-day change tracking mechanism to trace historical changes in VM availability for quick debugging. With these new additions, we’re excited to announce the general availability of VM availability information in the HealthResources dataset in ARG! With this offering users can:

Efficiently query the latest snapshot of VM availability across all Azure subscriptions at once and at low latencies for periodic and fleetwide monitoring.
Accurately assess the impact to fleetwide business SLAs and quickly trigger decisive mitigation actions, in response to disruptions and type of failure signature.
Set up custom dashboards to supervise the comprehensive health of applications by joining VM availability information with additional resource metadata present in ARG.
Track relevant changes in VM availability across a rolling 14-day window, by using the change-tracking mechanism for conducting detailed investigations.

Getting started

Users can query ARG via PowerShell, REST API, Azure CLI, or even the Azure Portal. The following steps detail how data can be accessed from Azure Portal.

Once on the Azure Portal, navigate to Resource Graph Explorer which will look like the below image:

Figure 1: Azure Resource Graph Explorer landing page on Azure Portal.

Select the Table tab and (single) click on the HealthResources table to retrieve the latest snapshot of VM availability information (availability state and health annotations).

Figure 2: Azure Resource Graph Explorer Window depicting the latest VM availability states and VM health annotations in the HealthResources table.

There will be two  types of events populated in the HealthResources table:

 
Figure 3: Snapshot of the type of events present in the HealthResources table, as shown in Resource Graph Explorer on the Azure Portal.

microsoft.resourcehealth/availabilitystatuses

This event denotes the latest availability status of a VM, based on the health checks performed by the underlying Azure platform. Below are the availability states we currently emit for VMs:

Available: The VM is up and running as expected.
Unavailable: We’ve detected disruptions to the normal functioning of the VM and therefore applications will not run as expected.
Unknown: The platform is unable to accurately detect the health of the VM. Users can usually check back in a few minutes for an updated state.

To poll the latest VM availability state, refer to the properties field which contains the below details:

Sample

{
      "targetResourceType": "Microsoft.Compute/virtualMachines",
      "previousAvailabilityState": "Available",
"targetResourceId": "/subscriptions/<subscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Compute/virtualMachines/<VMName>",
      "occurredTime": "2022-10-11T11:13:59.9570000Z",
      "availabilityState": "Unavailable"
}

Property descriptions

Field

Description

Corresponding RHC field

targetResourceType

Type of resource for which health data is flowing

resourceType

targetResourceId

Resource Id

resourceId

occurredTime

Timestamp when the latest availability state is emitted by the platform

eventTimestamp

previousAvailabilityState

Previous availability state of the VM

previousHealthStatus

availabilityState

Current availability state of the VM

currentHealthStatus

Refer to this doc for a list of starter queries to further explore this data.

microsoft.resourcehealth/resourceannotations (NEWLY ADDED)

This event contextualizes any changes to VM availability, by detailing necessary failure attributes to help users investigate and mitigate the disruption as needed. See the full list of VM health annotations emitted by the platform.
These annotations can be broadly classified into three buckets:

Downtime Annotations: These annotations are emitted when the platform detects VM availability transitioning to Unavailable. (For example, during unexpected host crashes, rebootful repair operations).
Informational Annotations: These annotations are emitted during control plane activities with no impact to VM availability. (Such as VM allocation/Stop/Delete/Start). Usually, no additional customer action is required in response.
Degraded Annotations: These annotations are emitted when VM availability is detected to be at risk. (For example, when failure prediction models predict a degraded hardware component that can cause the VM to reboot at any given time). We strongly urge users to redeploy by the deadline specified in the annotation message, to avoid any unanticipated loss of data or downtime.

To poll the associated VM health annotations for a resource, if any, refer to the properties field which contains the following details:

Sample

{
     "targetResourceType": "Microsoft.Compute/virtualMachines",                                                                                                                                                                        "targetResourceId": "/subscriptions/<subscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Compute/virtualMachines/<VMName>",
     "annotationName": "VirtualMachineHostRebootedForRepair",
     "occurredTime": "2022-09-25T20:21:37.5280000Z",
     "category": "Unplanned",
     "summary": "We're sorry, your virtual machine isn't available because an unexpected failure on the host server. Azure has begun the auto-recovery process and is currently rebooting the host server. No  additional action is required from you at this time. The virtual machine will be back online after the reboot completes.",
     "context": "Platform Initiated",
     "reason": "Unexpected host failure"
}

Property descriptions

Field

Description

Corresponding RHC field

targetResourceType

Type of resource for which health data is flowing

resourceType

targetResourceId

Resource Id

resourceId

occurredTime

Timestamp when the latest availability state is emitted by the platform

eventTimestamp

annotationName

Name of the Annotation emitted

eventName

reason

Brief overview of the availability impact observed by the customer

title

category

Denotes whether the platform activity triggering the annotation was either planned maintenance or unplanned repair. This field is not applicable to customer/VM-initiated events.

Possible values: Planned | Unplanned | Not Applicable | Null

category

context

Denotes whether the activity triggering the annotation was due to an authorized user or process (customer-initiated), or due to the Azure platform (platform-initiated) or even activity in the guest OS that has resulted in availability impact (VM initiated).

Possible values: Platform-initiated | User-initiated | VM-initiated | Not Applicable | Null

context

summary

Statement detailing the cause for annotation emission, along with remediation steps that can be taken by users

summary

Refer to this doc for a list of starter queries to further explore this data.

Looking ahead to 2023, we have multiple enhancements planned for the annotation metadata that is surfaced in the HealthResources dataset. These enrichments will give users access to richer failure attributes to decisively prepare a response to a disruption. In parallel, we aim to extend the duration of historical lookback to a minimum of 30 days so users can comprehensively track past changes in VM availability.

VM availability metric in Azure Monitor Preview

We’re excited to share that the out-of-box VM availability metric is now available as a public preview for all users! This metric displays the trend of VM availability over time, so users can:
Set up threshold-based metric alerts on dipping VM availability to quickly trigger appropriate mitigation actions.
Correlate the VM availability metric with existing platform metrics like memory, network, or disk for deeper insights into concerning changes that impact the overall performance of workloads.
Easily interact with and chart metric data during any relevant time window on Metrics Explorer, for quick and easy debugging.
Route metrics to downstream tooling like Grafana dashboards, for constructing custom visualizations and dashboards.

Getting started

Users can either consume the metric programmatically via the Azure Monitor REST API or directly from the Azure Portal. The following  steps highlight metric consumption from the Azure Portal.

Once on the Azure Portal, navigate to the VM overview blade. The new metric will display as VM Availability (Preview), along with other platform metrics under the Monitoring tab.

Figure 4: View the newly added VM Availability Metric on the VM overview page on Azure Portal.

Select (single click) the VM availability metric chart on the overview page, to navigate to Metrics Explorer for further analysis.

Figure 5: View the newly added VM availability Metric on Metrics Explorer on Azure Portal.

Metric description:

Display Name

VM Availability (preview)

Metric Values

1 during expected behavior; corresponds to VM in Available state.

0 when VM is impacted by rebootful disruptions; corresponds to VM in Unavailable state.

NULL (shows a dotted or dashed line on charts) when the Azure service that is emitting the metric is down or is unaware of the exact status of the VM; corresponds to VM in Unknown state.

Aggregation

The default aggregation of the metric is Average, for prioritized investigations based on extent of downtime incurred.

The other aggregations available are:

Min, to immediately pinpoint to all the times where VM was unavailable.

Max, to immediately pinpoint to all the instances where VM was Available.

Refer here for more details on chart range, granularity, and data aggregation.

Data Retention

Data for the VM availability metric will be stored for 93 days to assist in trend analysis and historical lookback.

Pricing

Please refer to the Pricing breakdown, specifically in the “Metrics” and “Alert Rules” sections.

Looking ahead to 2023, we plan to include impact details (user vs platform initiated, planned vs unplanned) as dimensions to the metric, so users are well equipped to interpret dips, and set up much more targeted metric alerts. With the emission of dimensions in 2023, we also anticipate transitioning the offering to a general availability status.

Introducing instantaneous notifications on changes in VM availability via Event Grid

We’re thrilled to introduce our latest monitoring offering—the private preview of VM availability status change events in an Event Grid System Topic, which uses the low-latency technology of Azure Event Grid! Users can now subscribe to the system topic and route these events to their downstream tooling using any of the available event handlers (such as Azure Functions, Logic Apps, Event Hubs, and Storage queues). This solution uses an event-driven architecture to communicate scoped changes in VM availability to end users in less than five seconds from the disruption occurrence. This empowers users to take instantaneous mitigation actions to prevent end user impact.

As part of the private preview, we’ll emit events scoped to changes in VM availability states, with the sample schema below:

Sample

{
     "id": "4c70abbc-4aeb-4cac-b0eb-ccf06c7cd102",
     "topic": "/subscriptions/<subscriptionId>,
   "subject": "/subscriptions/<subscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Compute/virtualMachines/<VMName>/providers/Microsoft.ResourceHealth/AvailabilityStatuses/current",
    "data": {
        "resourceInfo": {
"id":"/subscriptions/<subscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Compute/virtualMachines/<VMName>/providers/Microsoft.ResourceHealth/AvailabilityStatuses/current",       
"properties": {
"targetResourceId":"/subscriptions/<subscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.Compute/virtualMachines/<VMName>"
              "targetResourceType": "Microsoft.Compute/virtualMachines",
              "occurredTime": "2022-09-25T20:21:37.5280000Z"
"previousAvailabilityState": "Available",
      "availabilityState": "Unavailable"
           }
        },
        "apiVersion": "2020-09-01"
     },
"eventType": "Microsoft.ResourceNotifications.HealthResources.AvailabilityStatusesChanged",
   "dataVersion": "1",
     "metadataVersion": "1",
     "eventTime": "2022-09-25T20:21:37.5280000Z"
}

The properties field is fully consistent with the microsoft.resourcehealth/availabilitystatuses event in ARG. The event grid solution offers near-real-time alerting capabilities on the data present in ARG.

We’re currently releasing the preview to a small subset of users to rigorously test the solution and collect iterative feedback. This approach enables us to preview and even announce the general availability of a high quality and well-rounded offering in 2023. As we look towards the general availability of this solution, users can expect to receive events when annotations, automated RCAs are emitted by the platform.

What’s next?

We’ll be heavily focused on strengthening our monitoring platform to continuously improve the experience for customers based on ongoing feedback collected from the community (such as  aggregated VMSS health showing degraded inaccurately, VM unavailable for 15 minutes, Missing VM downtimes in Activity Log). By streamlining our internal message pipeline, we aim to not only improve data quality, but also maintain data consistency across our offerings and expand the scope of failure scenarios surfaced.

Introducing Degraded VM Availability state

In light of our upcoming efforts to centralize our monitoring architecture, we’ll be well-positioned to introduce a Degraded VM availability state for virtual machines in 2023. This state will be extremely useful in setting up targeted alerts on predicted hardware failure scenarios where there is imminent risk to VM availability. This state will also allow users to efficiently track cases of degraded hardware or software failures needing to redeploy, which today do not cause a corresponding change in VM availability. We will also aim to emit reminder annotations through the duration of the VM being marked Degraded, to prevent users from overlooking the request to redeploy.

Expand scope of failure attribution to include application freeze events

In 2023, we plan to expand our scope of failure attribution and emission to also include application freeze events that may be caused due to network agent updates, host OS updates lasting thirty seconds and freeze-causing repair operations. This will ensure users have enhanced visibility into freeze impact and will be applied across our monitoring offerings, including Resource Health and Activity Logs.

Learn More

Please stay tuned for more announcements on the Flash initiative, by tracking updates to the Advancing Reliability Series!
Quelle: Azure

Do more with less using new Azure HX and HBv4 virtual machines for HPC

This post was co-authored by Jyothi Venkatesh, Senior Product Manager, Azure HPC and Fanny Ou, Technical Program Manager, Azure HPC.

The next generation of purpose-built Azure HPC virtual machines

Today, we are excited to announce two new virtual machines (VMs) that deliver more performance, value-adding innovation, and cost-effectiveness to every Azure HPC customer. The all-new HX-series and HBv4-series VMs are coming soon to the East US region, and thereafter to the South Central US, West US3, and West Europe regions. These new VMs are optimized for a variety of HPC workloads such as computational fluid dynamics (CFD), finite element analysis, frontend and backend electronic design automation (EDA), rendering, molecular dynamics, computational geoscience, weather simulation, AI inference, and financial risk analysis.

Innovative technologies to help HPC customers where it matters most

HX and HBv4 VMs are packed with new and innovative technologies that maximize performance and minimize total HPC spend, including:

4th Gen AMD EPYC™ processors (Preview, Q4 2022).
Upcoming AMD EPYC processors, codenamed "Genoa-X," (with general availability in 1H 2023).
800 GB/s of DDR5 memory bandwidth (STREAM TRIAD).
400 Gb/s NVIDIA Quantum-2 CX7 InfiniBand, the first on the public cloud.
80 Gb/s Azure Accelerated Networking.
PCIe Gen4 NVMe SSDs delivering 12 GB/s (read) and 7 GB/s (write) of storage bandwidth.

Below are preliminary benchmarks from the preview of HBv4 and HX series VMs using 4th Gen AMD EPYC processors across several common HPC applications and domains. For comparison, performance information is also included from Azure’s most recent H-series (HBv3-series with Milan-X processors), as well as a 4-year-old HPC-optimized server commonly found in many on-premises datacenters (represented here by Azure HC-series with Skylake processors).

Figure 1: Performance comparison of HBv4/HX-series in Preview to HBv3-series and four-year-old server technology in an HPC-optimized configuration across diverse workloads and scientific domains.

Learn more about the performance of HBv4 and HX-series VMs with 4th Gen EPYC CPUs.

HBv4-series brings performance leaps across a diverse set of HPC workloads

Azure HBv3 VMs with 3rd Gen AMD EPYC™ processors with AMD 3D V-cache™ Technology already deliver impressive levels of HPC performance, scaling MPI workloads up to 27x higher than other clouds, surpassing many of the leading supercomputers in the world, and offering the disruptive value proposition of faster time to solution with lower total cost. Unsurprisingly, the response from customers and partners has been phenomenal. With the introduction of HBv4 series VMs, Azure is raising the bar yet again—this time across an even greater diversity of memory performance-bound, compute-bound, and massively parallel workloads.

VM Size

Physical CPU Cores

RAM (GB)

Memory Bandwidth (STREAM TRIAD) (GB/s)

L3 Cache/VM (MB)

FP64 Compute (TFLOPS)

InfiniBand RDMA Network (Gbps)

Standard_HB176rs_v4

176

688

800

768 MB

6

400

Standard_HB176-144rs_v4

144

688

800

768 MB

6

400

Standard_HB176-96rs_v4

96

688

800

768 MB

6

400

Standard_HB176-48rs_v4

48

688

800

768 MB

6

400

Standard_HB176-24rs_v4

24

688

800

768 MB

6

400

Notes: 1) “r” denotes support for remote direct memory access (RDMA) and “s” denotes support for Premium SSD disks. 2) At General Availability, Azure HBv4 VMs will be upgraded to Genao-X processors featuring 3D V-cache. Updated technical specifications for HBv4 will be posted at that time.

HX-series powers next generation silicon design

In Azure, we strive to deliver the best platform for silicon design, both now and far into the future. Azure HBv3 VMs, featuring 3rd Gen AMD EPYC processors with AMD 3D V-cache Technology, are a significant step toward this objective, offering the highest performance and total cost effectiveness in the public cloud for small and medium memory EDA workloads. With the introduction of HX-series VMs, Azure is enhancing its differentiation with a VM purpose-built for even larger models becoming commonplace among chip designers targeting 3, 4, and 5 nanometer processes.

HX VMs will feature 3x more RAM than any prior H-series VM, up to nearly 60 GB of RAM per core, and constrained cores VM sizes to help silicon design customers maximize ROI of their per-core commercial licensing investments.

VM Size

Physical CPU Cores

RAM (GB)

Memory/Core(GB)

L3 Cache/VM (MB)

Local SSD NVMe (TB)

InfiniBand RDMA Network (Gbps)

Standard_HX176rs

176

1,408

8

768

3.6 TB

400

Standard_HX176-144rs

144

1,408

10

768

3.6 TB

400

Standard_HX176-96rs

96

1,408

15

768

3.6 TB

400

Standard_HX176-48rs

48

1,408

29

768

3.6 TB

400

Standard_HX176-24rs

24

1,408

59

768

3.6 TB

400

Notes: 1) “r” denotes support for remote direct memory access (RDMA) and “s” denotes support for Premium SSD disks. 2) At General Availability, Azure HBv4 VMs will be upgraded to Genoa-X processors featuring 3D V-cache. Updated technical specifications for HBv4 will be posted at that time.

400 Gigabit InfiniBand for supercomputing customers

HBv4 and HX VMs are Azure’s first to leverage 400 Gigabit NVIDIA Quantum-2 InfiniBand. This newest generation of InfiniBand brings greater support for the offload of MPI collectives, enhanced congestion control, and enhanced adaptive routing capabilities. Using the new HBv4 or HX-series VMs and only a standard Azure Virtual Machine Scale Set (VMSS), customers can scale CPU-based MPI workloads beyond 50,000 cores per job.

Continuous improvement for Azure HPC customers

Microsoft and AMD share a vision for a new era of high-performance computing in the cloud: one defined by constant improvements to the critical research and business workloads that matter most to our customers. Azure continues to collaborate with AMD to make this vision a reality by raising the bar on the performance, scalability, and value we deliver with every release of Azure H-series VMs.

Figure 2: Azure HPC Performance 2019 through 2022.

Learn more about the performance of HBv4 and HX-series VMs with 4th Gen EPYC CPUs.

Customer and partner momentum

"We’re pleased to see Altair® AcuSolve®’s impressive linear scale-up on the HBv3 instances, showing up to 2.5 times speedup. Performance increases 12.83 times with an 8-node (512-core) configuration on 3rd AMD EPYC™ processors, an excellent scale-up value for AcuSolve compared to the previous generation delivering superior price performance. We welcome the addition of the new Azure HBv4 and HX-series virtual machines and look forward to pairing them with Altair software to the benefit of our joint customers.”

—Dr. David Curry, Senior Vice President, CFD and EDEM

"Customers in the HPC industry continue to demand higher performance and optimizations to run their most mission-critical and data-intensive applications. 4th Gen AMD EPYC processors provide breakthrough performance for HPC in the cloud, delivering impressive time to results for customers adopting Azure HX-series and HBv4-series VMs."

—Lynn Comp, Corporate Vice President, Cloud Business, AMD

"Ansys electronics, semiconductor, fluids, and structures customers demand more throughput out of their simulation tools to overcome challenges posed by product complexity and project timelines. Microsoft's HBv3 virtual machines, featuring AMD’s 3rd Gen EPYC processors with 3D V-Cache, have been giving companies a great price/performance crossover point to support these multiphysics simulations on-demand and with very little IT overhead. We look forward to leveraging Azure’s next generation of HPC VMs featuring 4th Gen AMD EPYC processors, the HX and HBv4 series, to enable even greater simulation complexity and speed to help engineers reduce risk and meet time-to-market deadlines."

—John Lee, Vice President and General Manager, Electronics and Semiconductor, Ansys

"We’ve helped thousands of customers combine the performance and scalability of the cloud, providing ease-of-use and instance access to our powerful computational software, which speeds the delivery of innovative designs. The two new high-performance computing virtual machines powered by the AMD Genoa processor on Microsoft Azure can provide our mutual customers with optimal performance as they tackle the ever-increasing demands of compute and memory capacity for gigascale, advanced-node designs."

—Mahesh Turaga, Vice President, Cloud Business Development, Cadence

"Hexagon simulation software powers some of the most advanced engineering in the world. We’re proud to partner with Microsoft, and excited to pair our software with Azure’s new HBv4 virtual machines. During early testing in collaboration with the Azure HPC team, we have seen a generational performance speedups of 400 percent when comparing structural simulations running on HBv3 and HX-series VMs. We look forward to seeing what our joint customers will do with this remarkable combination of software and hardware to advance their research and productivity, now and tomorrow. In the first quarter of 2023, we will be benchmarking heavy industrial CFD computations, leveraging multiple HBv4 virtual machines connected through InfiniBand."

—Bruce Engelmann, CTO, Hexagon

"Microsoft Azure has once again raised the bar for HPC infrastructure platform in the cloud this time with the launch of Azure HBv4 and HX virtual machines based on AMD’s 4th gen EPYC Genoa CPUs. We are expecting a strong customer demand for HBv4 and are excited to offer it to our customers that would like to run CFD, EDA, or other types of HPC workloads in the cloud.

—Mulyanto Poort, Vice President of HPC Engineering at Rescale

"Early testing by AMD with Siemens EDA workloads showed 15 percent to 22 percent improvements in runtimes with Microsoft Azure’s new AMD-based virtual machines compared to the previous generation. Semiconductor chip designers face a range of technical challenges that make hitting release dates extremely difficult. The combined innovation of AMD, Microsoft Azure, and Siemens provides a simplified path to schedule predictability through the increased performance possible with the latest offerings."

—Craig Johnson, Vice President, Siemens, EDA Cloud Solutions

"Customer adoption of the cloud for chip development is accelerating, driven by complexity and time-to-market advantages. The close collaboration between Synopsys and Microsoft brings together EDA and optimized compute to enable customers to scale under the Synopsys FlexEDA pay-per-use model. Verification represents a significant EDA workload in today’s complex SoCs and with the release of AMD’s next-generation EPYC processor available on Microsoft Azure, customers can take advantage of the optimized cache utilization and NUMA-aware memory layout techniques to achieve up to 2x verification throughput over previous generations."

—Sandeep Mehndiratta, Vice President of Cloud at Synopsys

Learn more

Sign up to request access to the new VMs.
NEW Azure HPC + AI Tech Community Blog.
Performance and Scalability of HBv4 and HX VMs.
Learn more about Azure HPC + AI.

#AzureHPCAI
Quelle: Azure