This first article in our resilience series draws on a conversation with Mark Russinovich about how resilience is changing in the AI era and what it takes to continuously validate it at scale.
What surprises me most about resilience failures is how ordinary the drift is. A workload is deployed across availability zones, but a health probe still points to a single dependency. A database supports failover, but the application’s connection string is pinned to one region. Nothing looks broken. The architecture diagram still shows a resilient design even as the operational reality underneath it changes.
For years, resilience was something you set up once: configure disaster recovery, write a runbook, run the occasional failover test. That kept the lights on, but it treated resilience as a project with an end date rather than a property you maintain.
So, when an availability zone or a region has a bad day, the question is not whether a recovery plan exists on paper. It is whether resilience is still true today, and whether the team can prove it.
The dependency that breaks a workload is also changing. On Microsoft’s FY26 Q4 earnings call, Satya Nadella put it plainly: “For whatever reason, if a given model goes away, then you can’t be left high and dry. You need to be able to still continue your cyber operations.”
Traditional disaster recovery planning assumes the critical dependency is infrastructure. Increasingly it is an AI model, an inference endpoint, a retrieval pipeline, or a service operating under capacity constraints. A workload can be perfectly healthy from an infrastructure perspective and still fail its users because that dependency is unavailable, throttled, or economically impractical to run. That dependency rarely appears on the diagram at all.
There is a second shift underneath the first. An architecture diagram assumes a human drew it and a human will read it. Both of those assumptions are ending, and the dependencies it describes are no longer all deterministic. That changes what it means to know your estate is resilient.
This is the first in a series on how we are helping customers move to resilience that is designed in, measured as the estate changes, and improved over time. We have written before about how to design a resilient workload. This is a different problem: knowing whether hundreds of workloads still match their design today and being able to prove it. It is where a large share of our roadmap investment is now going.
Why resilience drifts
Resilience has always been a shared responsibility. We provide the availability zones, a secondary region of choice, and the replication primitives needed to support resiliency, and those do not drift. What drifts is the other half of the bargain: whether a given workload still uses them the way it was designed to, after a year of changes nobody flagged as risky. Change is where this concentrates. Across the industry, roughly 70 percent of cloud outages are related to change in some way—not dramatic failures, but ordinary modifications whose blast radius nobody re-evaluated.
That is why change discipline matters as much as design. Internally, a change rolls out to a canary region first, then a pilot region, with bake times where health signals are watched before it goes any further. That practice came out of an incident of our own, and it is the same discipline the Well-Architected Framework describes as safe deployment practices.
Learn more about the Azure Well-Architected Framework
Disaster recovery is reactive by design. Teams set recovery objectives when a project ships, stand up replication, and then move on, with little ongoing visibility into whether those objectives still hold as the workload changes. Resilience is designed once and rarely revisited. Across a growing estate of availability zones and regions, the distance between the resilience that was designed and the resilience that actually exists widens quietly, and it usually surfaces only during an incident.
const currentTheme =
localStorage.getItem(‘blogInABoxCurrentTheme’) ||
(window.matchMedia(‘(prefers-color-scheme: dark)’).matches ? ‘dark’ : ‘light’);
// Modify player theme based on localStorage value.
let options = {“autoplay”:false,”hideControls”:null,”language”:”en-us”,”loop”:false,”partnerName”:”cloud-blogs”,”poster”:”https://cdn-dynmedia-1.microsoft.com/is/image/microsoftcorp/1117657-resilience-drift_tbmnl_en-us?wid=1280″,”title”:”Resilience drift”,”sources”:[{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-resilience-drift-0x1080-6439k”,”type”:”video/mp4″,”quality”:”HQ”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-resilience-drift-0x720-3266k”,”type”:”video/mp4″,”quality”:”HD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-resilience-drift-0x540-2160k”,”type”:”video/mp4″,”quality”:”SD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-resilience-drift-0x360-958k”,”type”:”video/mp4″,”quality”:”LO”}],”ccFiles”:[{“url”:”https://azure.microsoft.com/en-us/blog/wp-json/bloginabox/v1/get-captions?url=https%3A%2F%2Fwww.microsoft.com%2Fcontent%2Fdam%2Fmicrosoft%2Fbade%2Fvideos%2Fproducts-and-services%2Fen-us%2Fazure%2F1117657-resilience-drift%2F1117657-resilience-drift_cc_en-us.ttml”,”locale”:”en-us”,”ccType”:”TTML”}],”downloadableFiles”:[{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-resilience-drift_transcript_en-us”,”locale”:”en-us”,”mediaType”:”transcript”}]};
if (currentTheme) {
options.playButtonTheme = currentTheme;
}
document.addEventListener(‘DOMContentLoaded’, () => {
ump(“ump-6ab4e1548fa53″, options);
});
We learned this the hard way. Mark tells the story of the storage change that came close to taking Azure down, and what it permanently changed about how we deploy.
What a diagram can’t tell you
A diagram is a claim about a system, made once, by someone reasoning about the system as they believed it to be. It is useful, and it is not evidence. There are four things it structurally cannot tell you.
Whether the goal is being met right now. A diagram has no timestamp. Health modeling does: described in the Well-Architected Framework and now available through health models in Azure Monitor, it represents an application as a hierarchy of its components and the signals underneath them, so health is expressed in terms the business recognizes rather than as a wall of resource-level metrics. Paired with service level indicators, it answers the only question that matters during an incident: is this application meeting its objective right now? The standard that matters is not our own. A service is only healthy if the customer thinks it is healthy—we can believe it is fine, but if the customer is not seeing a healthy service, we have a problem.
What “resilient” even means for this application. On a diagram, resilient is an adjective. A resiliency goal makes it a threshold you either meet or miss and defining it at the level of the application rather than resource by resource is what makes the answer meaningful.
Whether the failover path actually works. Every diagram draws the arrow. Only a test proves it.
The resources nobody drew. A diagram shows what someone remembered. Generated Infrastructure-as-Code covers every resource in the application. The difference between those two sets is where drift begins—and increasingly the reader on the other end is an agent rather than a person, working from the outcome you asked for instead of the picture you drew.
This is how we run Azure. Rather than asking each team to declare what healthy means for their service, we standardized on service level indicators, then applied machine learning to observed behavior so that healthy is defined by what the service actually does rather than by what someone assumed it would do.
This is the part I find most interesting. We wrote our documentation, our schemas, and our templates for people. Increasingly the thing reading them is an agent, working from the outcome you asked for rather than the picture you drew, and generating the resources itself. When the author and the reader are both machines, a diagram is no longer even the medium the decision is made in.
const currentTheme =
localStorage.getItem(‘blogInABoxCurrentTheme’) ||
(window.matchMedia(‘(prefers-color-scheme: dark)’).matches ? ‘dark’ : ‘light’);
// Modify player theme based on localStorage value.
let options = {“autoplay”:false,”hideControls”:null,”language”:”en-us”,”loop”:false,”partnerName”:”cloud-blogs”,”poster”:”https://cdn-dynmedia-1.microsoft.com/is/image/microsoftcorp/1117657-proving-resilience_tbmnl_en-us?wid=1280″,”title”:”Proving resilience”,”sources”:[{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience-0x1080-6439k”,”type”:”video/mp4″,”quality”:”HQ”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience-0x720-3266k”,”type”:”video/mp4″,”quality”:”HD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience-0x540-2160k”,”type”:”video/mp4″,”quality”:”SD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience-0x360-958k”,”type”:”video/mp4″,”quality”:”LO”}],”ccFiles”:[{“url”:”https://azure.microsoft.com/en-us/blog/wp-json/bloginabox/v1/get-captions?url=https%3A%2F%2Fwww.microsoft.com%2Fcontent%2Fdam%2Fmicrosoft%2Fbade%2Fvideos%2Fproducts-and-services%2Fen-us%2Fazure%2F1117657-proving-resilience%2F1117657-proving-resilience_cc_en-us.ttml”,”locale”:”en-us”,”ccType”:”TTML”}],”downloadableFiles”:[{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience_transcript_en-us”,”locale”:”en-us”,”mediaType”:”transcript”},{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-proving-resilience_audio_en-us”,”locale”:”en-us”,”mediaType”:”audio”}]};
if (currentTheme) {
options.playButtonTheme = currentTheme;
}
document.addEventListener(‘DOMContentLoaded’, () => {
ump(“ump-6ab4e1548ff32″, options);
});
Mark on why we break our own services on purpose, what a game day actually tests, and why every failure mode we simulate is one that has really happened.
When the dependency is probabilistic
A model that disappears is the obvious risk. The subtler one is a model that answers, differently each time. It is a newer source of drift, and it does not behave like the old ones. Ask the same question of a model twice and you can get two different answers. That makes correctness harder to define, and it makes change harder to reason about: if you change the prompt, change the model, or change the harness and the skills around it, you have changed the software, and it deserves the same discipline as any other change.
Most teams smoke-test instead. It looks fine, so it ships. Evaluation is the step that gets skipped, and it is the one that matters: measuring the system against what you actually consider valuable, not simply whether it responded.
The first question is whether the system needs to be probabilistic at all. Stay as deterministic as you can and use AI where it earns its place, not everywhere. And where you can wrap a non-deterministic system in a deterministic check, do it. If an agent is only supposed to update dependency versions, have a second system verify that versions are the only thing that changed. Where a deterministic check is not possible, use adversarial review: a second agent whose whole job is to find what is wrong with the first one’s work.
And accountability does not transfer. When someone deploys an agent, someone remains answerable for what it does.
That applies to us as well. Our own incident triage system uses language models to answer which service is responsible for an incident—work that used to mean waking people up to argue over logs. It is genuinely faster, and it is still trust but verify.
const currentTheme =
localStorage.getItem(‘blogInABoxCurrentTheme’) ||
(window.matchMedia(‘(prefers-color-scheme: dark)’).matches ? ‘dark’ : ‘light’);
// Modify player theme based on localStorage value.
let options = {“autoplay”:false,”hideControls”:null,”language”:”en-us”,”loop”:false,”partnerName”:”cloud-blogs”,”poster”:”https://cdn-dynmedia-1.microsoft.com/is/image/microsoftcorp/1117657-ai-uncertainty_tbmnl_en-us?wid=1280″,”title”:”AI uncertainty”,”sources”:[{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty-0x1080-6439k”,”type”:”video/mp4″,”quality”:”HQ”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty-0x720-3266k”,”type”:”video/mp4″,”quality”:”HD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty-0x540-2160k”,”type”:”video/mp4″,”quality”:”SD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty-0x360-958k”,”type”:”video/mp4″,”quality”:”LO”}],”ccFiles”:[{“url”:”https://azure.microsoft.com/en-us/blog/wp-json/bloginabox/v1/get-captions?url=https%3A%2F%2Fwww.microsoft.com%2Fcontent%2Fdam%2Fmicrosoft%2Fbade%2Fvideos%2Fproducts-and-services%2Fen-us%2Fazure%2F1117657-ai-uncertainty%2F1117657-ai-uncertainty_cc_en-us.ttml”,”locale”:”en-us”,”ccType”:”TTML”}],”downloadableFiles”:[{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty_transcript_en-us”,”locale”:”en-us”,”mediaType”:”transcript”},{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-ai-uncertainty_audio_en-us”,”locale”:”en-us”,”mediaType”:”audio”}]};
if (currentTheme) {
options.playButtonTheme = currentTheme;
}
document.addEventListener(‘DOMContentLoaded’, () => {
ump(“ump-6ab4e154902b2″, options);
});
Mark on why guardrails around AI should be deterministic wherever possible, and why the agent is never the one accountable.
What this looks like in practice
That discipline has to hold wherever a workload runs, and the foundations are familiar. The reliability guidance in the Well-Architected Framework already says to design resilience in from the start, and the Cloud Adoption Framework describes the operating half: carrying that intent into how the estate is actually run and reviewed month after month. Where an estate spans global, national, and sovereign or regulated environments, the word resilient has to mean the same thing in each rather than being redefined at every boundary; our guidance on reliability and sovereignty covers where those constraints intersect.
Your diagram shows three zones. It does not show that the health probe behind the load balancer resolves to one of them. Availability zones protect a workload from datacenter-level failures within a region, but only if compute, storage, and data tiers are genuinely spread across them.
Plan region resiliency for disaster recovery. Set explicit recovery objectives, an RTO and RPO, for each workload. Availability zones give you high availability inside a region, while a secondary region of choice provides a failover location when an entire region is affected. They answer different risks, and a resilient design is deliberate about both. In a regulated estate the choice narrows further, because a recovery region has to sit inside the same jurisdiction as the workload it protects.
Decide how much resilience is worth buying. Resilience is a cost decision as much as a design one. An application carrying 100 million dollars of revenue on a single day justifies an active-active topology across regions for that day; the same application may run in a single region with active-passive failover for the rest of the year. The right answer is deliberate, not maximal.
Know your blast radius. Understand what each workload depends on and where its hidden single points of failure are, then keep that picture current as the application changes. Internally we do this as reliability threat modeling: the same discipline as security threat modeling, asking of each component what would happen if it failed, and what we would do about it.
Check the dependencies your recovery path itself relies on. A workload can be replicated correctly and still be unrecoverable: if its encryption keys live only in the primary region, they are gone precisely when a region outage means you need them. Recovery paths have dependencies too, and they are rarely on the diagram.
Your diagram probably has no box for the AI dependency your application now relies on. Design for your application, workload, AI model, and service dependencies, not just infrastructure. Plan for graceful degradation and fallback so that if a critical dependency is deprecated, throttled, unavailable, or capacity-constrained, the workload continues to operate through an alternative path rather than failing outright.
This is not theoretical. Carne Group, one of Europe’s largest independent third-party asset managers with one trillion dollars under management, rebuilt its estate on Azure with infrastructure-as-code landing zones precisely so resilience would be reproducible rather than remembered. Because the definition lives in code, their team can stand up a duplicate site in another region and, as Carne Group’s global technology lead Stéphane Bebrone puts it, “even in the event of a worst-case scenario, we could be back up and running more or less in the same day.” They are working toward an active-passive topology across regions and plan to use Chaos Studio to verify those failover paths on a schedule rather than on an incident. Under DORA, they have to be able to prove it, not assert it.
Learn more about how Carne Group used Azure Site Recovery and Backup
Closing the gap between intent and reality
An architecture diagram is a statement of intent, not proof. It may show zone redundancy, regional failover, and protected dependencies, but only a test can determine whether those assumptions still hold. Assembling the underlying data into a clear view of application resiliency takes real effort and expertise. That is the customer’s half of the shared responsibility, and it is the gap Azure Infrastructure Resiliency Manager is built to help close. In public preview, its agentic-first experience helps teams start resilient, get resilient, and stay resilient.
Start resilient. Define what resilient means for an application, set explicit resiliency goals, and use the Resiliency Agent to generate resiliency-aware Infrastructure-as-Code up front, so a new workload starts resilient instead of being corrected months later. Service Groups help teams represent an application as a logical group of Azure resources in the portal, making it easier to manage resiliency posture at the application level rather than resource by resource.
Get resilient. See where the estate stands against the intended design, identify resources that were never zone resilient or stopped being so after a change, and prioritize the gaps that matter first. Rather than correlating findings across several tools, teams get recommendations and generated Infrastructure-as-Code for supported fixes, so remediation can move through a pull request instead of becoming a separate project.
Stay resilient. Validate the assumptions behind the recovery path before a real outage does it for you. For workloads where the customer manages the compute, such as virtual machines, a zone-down drill can simulate the loss of an availability zone and show what actually happens to the application. For other services, teams can use failover validation, product-specific recovery capabilities, and fault injection through Azure Chaos Studio to test the right failure modes.
We will be candid about the gaps too. Consistent, self-service resiliency assessment across every workload and environment is not finished work, and saying so matters. Resiliency improves when teams can see the gaps, measure them, and systematically close them over time. It is also work that never quite ends. As services harden the failure rate drops, but the failures that remain get rarer and stranger; you approach perfection asymptotically without ever arriving.
The bottom line: a diagram is not proof
Resilience is not a project you finish; it is a posture you maintain. The work is to design it in from the first architecture decision, then keep proving it as the estate changes, so drift is caught by a test rather than by an incident. All of it is anchored in the Azure Essentials frameworks, the Well-Architected Framework, and the Cloud Adoption Framework, so that resilient and sovereign carry the same meaning across every environment instead of being reinvented team by team.
None of this is a new framework. A diagram tells you what you intended; only a test tells you what you have. That was true when people drew the diagrams and read them. It matters more now that neither is reliably the case, and that some of what you depend on answers differently every time you ask. If resilience cannot be tested, it cannot be trusted.
In the next part of this series, we will look at how to measure that posture at scale.
const currentTheme =
localStorage.getItem(‘blogInABoxCurrentTheme’) ||
(window.matchMedia(‘(prefers-color-scheme: dark)’).matches ? ‘dark’ : ‘light’);
// Modify player theme based on localStorage value.
let options = {“autoplay”:false,”hideControls”:null,”language”:”en-us”,”loop”:false,”partnerName”:”cloud-blogs”,”poster”:”https://cdn-dynmedia-1.microsoft.com/is/image/microsoftcorp/1117657-full-interview_tbmnl_en-us?wid=1280″,”title”:”Full interview”,”sources”:[{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview-0x1080-6439k”,”type”:”video/mp4″,”quality”:”HQ”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview-0x720-3266k”,”type”:”video/mp4″,”quality”:”HD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview-0x540-2160k”,”type”:”video/mp4″,”quality”:”SD”},{“src”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview-0x360-958k”,”type”:”video/mp4″,”quality”:”LO”}],”ccFiles”:[{“url”:”https://azure.microsoft.com/en-us/blog/wp-json/bloginabox/v1/get-captions?url=https%3A%2F%2Fwww.microsoft.com%2Fcontent%2Fdam%2Fmicrosoft%2Fbade%2Fvideos%2Fproducts-and-services%2Fen-us%2Fazure%2F1117657-full-interview%2F1117657-full-interview_cc_en-us.ttml”,”locale”:”en-us”,”ccType”:”TTML”}],”downloadableFiles”:[{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview_transcript_en-us”,”locale”:”en-us”,”mediaType”:”transcript”},{“url”:”https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/1117657-full-interview_audio_en-us”,”locale”:”en-us”,”mediaType”:”audio”}]};
if (currentTheme) {
options.playButtonTheme = currentTheme;
}
document.addEventListener(‘DOMContentLoaded’, () => {
ump(“ump-6ab4e15490faf”, options);
});
The three clips above are drawn from a longer conversation covering the 2014 change that came close to taking Azure down, how we learned to measure health from the customer’s point of view, what breaks differently when a dependency is an AI model, and why the hardware in a datacenter fails every single day. Check out the full interview on the Azure Essentials YouTube channel.
Azure Essentials
Check out the full interview on the Azure Essentials YouTube channel.
Watch now
Resources
Cloud Resiliency
Azure Essentials
Azure Infrastructure Resiliency Manager
Reliability design principles—Azure Well-Architected Framework
Health modeling for workloads—Azure Well-Architected Framework
Health models in Azure Monitor (preview)
Cloud Adoption Framework for Azure
Azure Architecture Center
Reliability and Sovereignty in Azure
What’s new in the Resiliency service
Enable Zone Resiliency for Azure Workloads
Azure Chaos Studio documentation
Azure Site Recovery documentation
Azure Backup documentation
Carne Group boosts infrastructure and cyber resilience with Azure Site Recovery and Backup
Safe deployment practices—Azure Well-Architected Framework
Meet Brain: The AI system behind Azure reliability
Optimizing incident management with AIOps using the Triangle System
Proving application resilience on Azure with Chaos Studio
Advancing reliability—Azure Blog series
The post Your architecture diagram is not your resilience appeared first on Microsoft Azure Blog.
Quelle: Azure
