If you use Microsoft email for business or personal use, you may have noticed a small glitch on Thursday as the email service went down. After a while, it became clear that this wasn’t a little glitch.

At 6:04 p.m. Eastern on Thursday, November 12, Microsoft’s status account posted that it was returning a portion of affected servers to the service, that they remained stable so far, and that “users may experience a gradual recovery as their clients reconnect.” Outlook had been failing for work and school accounts since a little after noon. If you were refreshing that account from an IT desk, it read like the end of a long afternoon.

At 7:06 p.m. Microsoft posted again: “The servers we returned to service can become unresponsive again once they receive traffic.”

Servers Microsoft had just brought back were falling over again, and what knocked them down was everyone trying to reach them at once. By one outside measure, traffic to Exchange Online, the service behind Outlook at work, ran about 85% below normal for roughly 18 hours. Most business customers had email again by Friday morning, and Microsoft was still restoring the rest that afternoon.

Microsoft runs one of the largest clouds on the planet. So why couldn't it just turn Outlook back on? Because the way back up turned out to be harder than the way down.

Thursday afternoon

Kentik, a company that measures internet traffic, puts the start at about 12:35 p.m. Eastern. Microsoft’s first public post came at 12:54 p.m. and described trouble reaching Exchange Online mailboxes “through multiple connection methods in the Americas.” It wasn't confined to the Americas for long. CNN reported that Downdetector, the site where people go to complain about outages, logged reports from 74 countries and territories in about an hour.

In Britain, where at least one police force said it couldn’t receive online form submissions, CNN reported that police urged the public to call 101, the non-emergency number, if a problem was “time-critical but not an emergency.” The National Health Service told patients to attend their appointments as normal, Air India temporarily rerouted its email (its flights weren’t affected), and at least one company pushed back its quarterly earnings, according to CNBC. Email users, including myself, were confused because other parts of Microsoft communication services were working just fine. Teams kept working for chat and meetings, though anything inside it that depended on Exchange, like the calendar, could fail. That’s how I found out, over Teams chat with a colleague, that the email server was down for everyone.

On r/sysadmin, one administrator kept a running log of Microsoft's status updates, each followed by a deadpan “Translation” that added some much-appreciated entertainment to the situation, and one reply asked whether anyone had tried turning the entirety of Microsoft off and then on again. Canva posted a fake blue error screen announcing that some software company had run into a problem. On Hacker News, people noticed that Microsoft's own status page was throwing errors, and someone suggested the status page needed a status page.

How many people it reached

Microsoft hasn’t published a count of the people affected, so the available figures only bracket it. At the low end, Downdetector peaked at more than 130,000 reports about Outlook. It counts only people who went to a website to complain, and as Al Jazeera pointed out, “the true number is likely far higher.” At the other end sits Microsoft's customer base: more than 450 million paid commercial seats, the figure it reported in January, according to Office 365 for IT Pros. Only part of that base was hit, because this failure was centered on Exchange.

In between, Anderson Economic Group, a Michigan research firm, gave CNN an estimate: perhaps as many as 26 million users on Thursday and 51 million on Friday had their operations disrupted “in a way that caused an economic loss,” and the lost productivity could approach $7 billion. Its president, Patrick Anderson, was clear about the limits: “This is a starting point, with all the limitations on what we know. We don't know complete scale.”

Fewer servers, same crowd

Microsoft's first explanation that afternoon was “elevated load,” which sounds like a crowd showing up. At 2:31 p.m. it corrected that: “the elevated load isn't driven by an increase in traffic, but by a number of servers being unexpectedly unavailable; straining the remaining infrastructure.”

The servers that dropped out were failing what Microsoft calls automated health checks. A health check is a small, routine test a load balancer runs against each server: are you answering, and answering correctly? A server that fails gets pulled out of rotation, and its traffic goes to the ones that passed. Picture a grocery store the Wednesday before Thanksgiving, with a manager watching every checkout lane. A lane stops moving, the manager closes it, and its line gets sent elsewhere. Most of the time this is the system protecting you from a broken machine.

By 3:19 p.m. Microsoft had a new theory. It said the failed health checks “may originate from a specific component that's responsible for processing server health.” If that's right, the manager was closing lanes that were moving fine. Around 4 p.m. it said it had validated “an alternative service health check method,” and some servers came back.

Every lane that closes sends its shoppers to the lanes still open. On Thursday, the servers still in service had a weak point of their own: each connection from a phone or laptop begins with a secure handshake, the quick exchange that sets up encryption. CIO reported that the server process handling those handshakes was being overwhelmed, “causing the servers to stop processing requests.” At 5:09 p.m. Microsoft posted that “our actions taken thus far have not provided the anticipated relief,” and that “a core process on affected servers is becoming overwhelmed, causing capacity loss.”

An overwhelmed server answers health checks slowly or not at all, so it gets pulled too, and its share of the crowd moves to the next lane. Google’s site reliability engineers describe this spiral in the chapter of their book on cascading failures, including the case where “health-checking itself makes the service unhealthy.”

Every phone kept trying

The crowd also wasn’t standing still. Outlook on your phone doesn’t give up when it can’t connect. It tries again, and so does every laptop, tablet, sync client and mail server that has a message to deliver. I may be an outlier personally, but I have two computers, a phone, and two tablets, all configured to my Outlook account, so all of them are checking, and that’s just for one person’s account. Kentik’s data, as PPC Land reported it, showed devices opening up to 70% more connections to Exchange than expected, while responses coming back from Microsoft fell to about 27% of normal. Cloudflare saw requests to Exchange Online roughly triple, “likely clients and apps retrying,” in its words.

Marc Brooker, an engineer at Amazon Web Services, wrote the standard explainer on this in Amazon's Builders’ Library. His one-line summary is “Retries are ‘selfish.’” Each client that retries is doing the sensible thing for itself, and the sum of all those sensible decisions is extra load on a system that’s already struggling. He also warns that retries “can even delay recovery by keeping the load high long after the original issue is resolved.”

In Kentik’s chart, connection attempts peaked shortly after 1 p.m. Eastern, eased off, and then spiked again at about 7:30 p.m., right after Microsoft started returning servers. Doug Madory, Kentik’s director of internet analysis, wrote that this was “the retry pressure Microsoft described as overwhelming servers once they came back online.”

Back on, then down again

That second spike came less than half an hour after Microsoft's 7:06 p.m. warning that returned servers could fail again under traffic. Bringing a server back at this scale means reopening it into a crowd that has been building for hours: every phone that’s been retrying, every queued message, every user bounced off a sick machine. They all find the first healthy server at once. Engineers call this a thundering herd. In the store, it’s the manager reopening one lane on Thanksgiving Wednesday and watching every cart in the building turn toward it.

Google’s SRE book describes servers in this spot being “bombarded with an extremely high rate of requests” the moment they come back online, until they “fail almost immediately.” Microsoft's own 7:06 p.m. post said it was “temporarily limiting which types of traffic can reach the affected infrastructure.” At 10:13 p.m. it posted that its changes weren't “taking as expected,” and by Friday morning in Europe, Al Jazeera reported that servers Microsoft had restored were failing again.

In 2021, Nathan Bronson and three co-authors gave this pattern a name, metastable failure: a trigger “causes the system to enter a bad state that persists even when the trigger is removed,” and a feedback loop holds it there. Their first example of that loop is the one Kentik measured: “One of the most common failure-sustaining mechanisms is request retries.” Getting out, they wrote, “requires a strong corrective push, such as rebooting the system or dramatically reducing the load.”

A follow-up study at OSDI in 2022 went looking for these failures in the wild and found 22 of them across 11 organizations, four at Azure. Retries were the most common way they kept going, in more than half the incidents. The authors also noted that engineers’ first instinct, fixing the trigger, often doesn't help, because “once in the metastable failure state, the system cannot recover all by itself.”

Microsoft hasn't called this a metastable failure, and its report hasn't confirmed a retry loop. What the public record shows matches the pattern in those papers: a failure that outlived its first fix, servers that fell over again under returning traffic, and outside measurements of a retry surge that spiked when they came back.

When Facebook went dark in October 2021, its engineers wrote afterward that once the network was back, “flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic,” so they brought load back deliberately. Two months later, Amazon’s own write-up of a major outage in its Virginia region described a capacity-scaling change that set off “a large surge of connection activity,” which produced “even more connection attempts and retries” and “persistent congestion.”

Letting the crowd back in a few at a time

Microsoft’s way out was to meter the recovery. Through Thursday night and into Friday, it walled off part of its infrastructure, restarted it, let a little traffic in, and watched whether it held before letting in more. It also added virtual machines, computers it can start up in software on its own hardware, to absorb the surge.

At 4:24 a.m. Eastern on Friday it extended its email timeout to 48 hours, “to help preserve delayed mail.” Mail servers that can't deliver a message keep retrying for a set period, then give up and bounce it back to the sender as undeliverable. Stretching that window to two days meant your Thursday afternoon email would wait in line instead of coming back to you with an error.

Recovery then went region by region: parts of Asia-Pacific first, then Europe, then North America. Kentik saw Exchange traffic start climbing at about 6:20 a.m. Eastern on Friday, and by 8:25 a.m. it was running about 19% above normal as queued mail and mailbox syncs caught up. That spike is your Thursday email landing, along with everyone else's. At 9:17 a.m. Microsoft said email queues were “significantly reduced and approaching normal levels.”

At 1:05 p.m. on Friday, Microsoft explained the long tail in a statement that also apologized: “Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration.” The outage still wasn't over. By Friday evening The Register was counting past 27 hours.

What started it

That 1:05 p.m. statement also named a cause, briefly. Microsoft wrote that “The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change,” that it had “addressed the underlying issue,” and that this “was a technical issue not related to cyber or AI / agents.” It promised “a full post-incident report once our review is complete.”

The statement doesn't say what went wrong in the upgrade, and it sits awkwardly beside Thursday's posts. At 3:19 p.m., Microsoft said it had “not identified any recent change correlated with the start of impact.” About 80 minutes later it was assessing whether to revert “a small number of recent configuration updates.” By Friday, CIO reported it was reverting a network configuration change that “correlated with the start of the disruption.” The report will need to reconcile those three accounts.

Online, people had picked a cause well before Microsoft named one. The top comment on one large r/technology thread blamed AI-written code that nobody knew how to fix, which Microsoft’s statement now rules out. Others blamed layoffs, often with a figure of 60,000 jobs. The reporting I can find says Microsoft cut about 4,800 roles in July, about 2.1% of its workforce. It had also offered buyouts in April, to an undisclosed number of employees. Nothing I’ve found connects those cuts to this outage. Until the report is out, anyone telling you exactly why your email stopped on Thursday is guessing.

The case against blaming the crowd

Michael Grothaus made the strongest objection to this account in Fast Company on Friday morning: “Such a long service interruption is rare for a platform as critical for communications and as large as Outlook,” especially one run by “a nearly $4 trillion company with nearly unlimited technical resources at its disposal.” Read that way, the thundering herd sounds like an alibi, since by Microsoft's own account what broke Exchange was an issue in its network upgrade.

The research backs him on the trigger. The OSDI study found that around 45% of the triggers in its sample were “engineer errors, such as buggy configuration or code deployments.” Microsoft has seen one of its own fixes make things worse before, too. In January, Microsoft 365 went down across North America for eight to ten hours, depending on whose count you use. TechRepublic reported that an early load-balancing adjustment “actually worsened the situation by creating additional traffic imbalances.”

After two outages at Azure Front Door, which routes traffic into much of its cloud, Microsoft published its lessons in December 2025. It said incompatible or erroneous configuration should “never propagate beyond canary environments.” A canary is the small slice of the system that gets every change first, so a bad change breaks something small. Microsoft added longer waits between rollout stages and a one-click revert to the last configuration known to work. It also admitted that during one of those October rollouts, the health-check validations “were all passed” while the problem spread. Those commitments covered Front Door, and nothing I’ve found shows Microsoft made the same ones for Exchange.

If retries did help keep Exchange down, part of that herd was Microsoft’s own software, because Outlook on phones and laptops is Microsoft’s code. Amazon's 2021 write-up admits the same thing about its own clients: they “have well tested request back-off behaviors,” it said, but “a latent issue prevented these clients from adequately backing off during this event.” Whether Microsoft’s clients backed off properly on Thursday night is a question its report can answer, because it wrote the clients.

Bronson and his co-authors wrote in 2021 that “It is common for an outage that involves a metastable failure to be initially blamed on the trigger, but the true root cause is the sustaining effect." The trigger is why Outlook went down on Thursday afternoon. If the pattern holds, the retries, the pulled servers and the herd are why it stayed down until Friday.

How systems are built to survive a herd

On the server side, the main defense is load shedding: a swamped server turns new work away quickly and politely, telling the client to come back later, instead of accepting everything and collapsing. The OSDI authors found load shedding was “the most popular mitigation effort used in over 50% of the incidents” they studied, and also that it's hard to get right, because without understanding the feedback loop, “it is hard to know just how much the load needs to be reduced.” Google's SRE book recommends the slow start Microsoft eventually used: “Gradually ramping up the load. This strategy allows caches to warm up, connections to be established.”

On the client side, the defense is backoff with jitter. Each failed attempt waits a little longer than the last, and the wait includes some randomness, so millions of devices don’t all try again in the same second. Brooker’s article walks through both the backoff and the randomness. Without the randomness, clients that failed together retry together.

Your team's version of 101

Most of us don't run Exchange, and what we control is much smaller. Sanchit Vir Gogia, an analyst at Greyhound Research, told CIO that “Enterprises have priced Microsoft 365 as a licence and never as a single point of failure,” and that “The answer is not a second suite. It is a rehearsed fallback for the few things that cannot stop: an out-of-band channel for incident command, mail continuity for critical roles, and a written decision on whether each security control fails open or closed.”

I’ve said before that you don't own a business you can't switch back on yourself. Email is the exception nearly everyone accepts, since nobody is going to start running a mail server over one bad Thursday. What you can decide ahead of time is how the few things that can’t stop will keep going while email is down. Gogia’s three items make good questions for a team to answer in an afternoon, and the British police add a fourth:

  1. Where do we talk if email is down? Pick the channel, and make sure it doesn’t sign in through the same account that just failed.
  2. Which roles can't go a day without mail? Name them, and decide how they’ll send and receive if Outlook doesn't.
  3. When a security tool can't reach Microsoft, does it block everything or let everything through? That’s what “fails open or closed” means, and the time to choose is before the outage.
  4. What do customers do instead? Know the phone number or page you’ll point them to, and who posts it.

On Thursday afternoon, Britain’s police already had theirs: call 101.

The video version of this story is on YouTube. If this was useful, there’s more, including my latest book, at davesaunders.net.

Sources