Originally published on Naveed Ahmed's Engineering Blog.
On November 12–13, 2026, a widespread Microsoft 365 outage affected Outlook, Exchange Online, parts of Teams, Copilot, and other services.
Microsoft linked the disruption to problems following a network-capacity upgrade, with server availability and infrastructure load contributing to the incident.
For DevOps and SRE engineers, it's a useful example of why distributed systems need more than application-level redundancy.
The exact internal routing sequence described in some analyses has not been publicly established, so let's focus on the engineering lessons we can apply.
1. Shared Dependencies Can Defeat Redundancy
Imagine a service deployed across three availability zones.
It has multiple replicas and healthy application containers.
But every replica depends on the same external authentication service.
If that dependency becomes unavailable, your application can still fail.
The lesson is to identify shared-fate dependencies: networking, identity, DNS, control planes, and critical external APIs.
2. Canary Infrastructure Changes
Application code isn't the only thing that needs progressive deployment.
Network routing policies, identity configurations, and infrastructure changes should also be staged.
A safe approach includes:
- Pre-deployment validation
- Small canary groups
- Synthetic monitoring
- Clear rollback conditions
- Observation periods before wider rollout
The rollback process must be tested rather than assumed to work.
3. Synthetic Monitoring Matters
Component health checks aren't enough.
A server may be running while customers cannot complete a login or send a message.
Useful synthetic probes might test:
DNS resolution ↓ TLS connection ↓ Authentication ↓ Application request ↓ Expected response Track both component metrics and user-facing functionality.
4. Prevent Retry Storms During Recovery
When an upstream service recovers, many clients may reconnect simultaneously.
If retries happen on fixed intervals, they can produce synchronized traffic spikes.
For safe, idempotent requests, bounded exponential backoff with jitter is often useful:
import random def retry_delay(attempt, base=0.5, cap=30): ceiling = min(cap, base * (2 ** attempt)) return random.uniform(0, ceiling) A complete implementation also needs timeouts, a maximum attempt count, and careful handling of non-idempotent operations.
5. Build an Incident Response Runbook
A practical runbook should explain:
- How to detect user impact.
- How to identify the failing dependency.
- Who owns the incident.
- Which mitigations are safe.
- How customers receive updates.
- How recovery is verified.
When the cloud provider itself is affected, independent monitoring and out-of-band communication become particularly important.
Read the Full Microsoft Outage Analysis
My original article covers the reported incident, possible network failure propagation, the distinction between Microsoft 365 and Azure service layers, telemetry safeguards, and an SRE resilience runbook.
Read Microsoft Outage: Root Cause and SRE Runbook
The low-level BGP sequence and precise internal metrics discussed in the article should be treated as unverified architectural analysis pending a detailed public postmortem.
What's the most important shared dependency in your production architecture?
Written by Naveed Ahmed — DevOps & Platform Engineering
