How Companies Stay Online During AWS us-east-1 Outages: What the Teams That Stayed Up Did Differently

Companies that stayed online during AWS us-east-1 outages already ran their critical services in a second region or a second cloud, moved traffic with a small, rehearsed step or automated health-checked routing, and kept us-east-1-only services out of their failover path. Multi-AZ inside us-east-1 was not enough, because the October 2025 outage hit regional services that every availability zone depends on. On Control Plane, a platform that runs each workload active-active across regions and clouds, customers with us-east-1 in their deployment had workloads active and running in healthy locations within 10 seconds during that outage, according to Control Plane’s internal incident telemetry.

Disclosure: Control Plane publishes this guide.

TL;DR

• What happened. On October 19-20, 2025, a fault in DynamoDB’s automated DNS management in us-east-1 spread to EC2 instance launches, Network Load Balancer health checks, Lambda, and container services across the region.
• Why us-east-1 outages spread so far. It is AWS’s oldest region, many production workloads run there, and the control planes for several global AWS services run there, so changes to IAM, Route 53, and CloudFront can fail during a us-east-1 event even for workloads in other regions.
• What the teams that stayed up had in common. They replicated data before the incident, removed us-east-1-only services from the request path, and practiced failover in game days.
• What multi-AZ covers. Availability zones protect against zone and data center failures. They do not protect against region-wide service failures such as the October 2025 DynamoDB endpoint DNS failure.
• The readiness checklist. Twelve checks, from a dependency map and replicated secrets to measured replication lag and a failback plan.
• How Control Plane fits. Control Plane’s Global Virtual Cloud (GVC), a set of locations in one cloud or several where each workload runs, serves traffic from every location and routes requests to the nearest healthy one.

What happened in the AWS us-east-1 outages?

According to AWS’s post-event summary, a race condition in DynamoDB’s automated DNS management left an empty DNS record for the regional endpoint, dynamodb.us-east-1.amazonaws.com, starting at 11:48 PM PDT on October 19, 2025. DynamoDB recovered in the early hours of October 20. EC2 instance launches stayed impaired and Network Load Balancer health checks failed into the afternoon, which affected Lambda, ECS, EKS, Fargate, STS, and other services that depend on them. AWS reported full recovery at 2:20 PM PDT on October 20.

Control Plane’s Cloud Outage Tracker covers 62 major AWS, Azure, Google Cloud, and Cloudflare outages since 2020. Recent us-east-1 events in the tracker include:

The tracker lists 9 of AWS’s 14 major outages in us-east-1. The median major outage across all providers lasted 6h 08m.

Why does us-east-1 go down so often, and why does it take so much of the internet with it?

AWS does not publish incident rates by region, so there is no official comparison of how often us-east-1 fails against other regions. Three facts are documented. us-east-1 is AWS’s oldest region, launched in 2006, and many production workloads run there. The control planes for several global AWS services run there. In the Cloud Outage Tracker, 9 of AWS’s 14 major outages since 2020 were in us-east-1, a count that is not adjusted for region size.

The control plane dependencies explain most of the spread. According to AWS’s fault isolation guidance for global services, the control planes for IAM, Route 53, CloudFront, AWS Certificate Manager certificates for CloudFront, AWS Organizations, and AWS WAF for CloudFront run in us-east-1. During a us-east-1 event, creating an IAM role, changing a Route 53 record through the API, or issuing a CloudFront certificate can fail even for workloads in other regions. Since December 2025, Route 53 offers a standby control plane endpoint in us-west-2 for updating public records during a us-east-1 event, if you configure and use it.

Data plane operations usually keep working. Route 53 DNS answers are served from AWS’s global edge network, and Route 53 health checks run in the data plane. Requests signed with existing IAM credentials are authorized by IAM’s regional data plane, though AWS reported STS and IAM sign-in errors in us-east-1 during the October 2025 event. Applications that refresh credentials through the global STS endpoint depend on us-east-1, so configure regional STS endpoints. A failover procedure that needs a us-east-1 control plane call may fail during the outage it is meant to handle.

How did companies stay online during the us-east-1 outage?

Four engineering teams published accounts of the October 2025 outage. Each row below uses only facts from that team’s own post, plus Control Plane’s incident data.

Temporal and FireHydrant. A rehearsed failover is a short, predictable step. FireHydrant had replicated its Namespaces to a second region and tested the move in a game day, so when errors appeared it triggered the failover and its workflows kept running. Temporal’s account also shows the risk of hidden dependencies: a few auto-failovers were delayed because the failover workflow itself depended on the source region.

LoginRadius. Identity is a dependency for every other service a customer runs. LoginRadius kept its customer-facing APIs online by running stateless services in independent regions with routing at two layers.

Block. Regional diversity paid off where it existed. Cash App ran in us-west-2 and was mostly unaffected, and Square’s automated failover redirected all traffic within a half hour. European seller transactions, whose primary path ran through us-east-1, still saw up to 30% of transactions fail at peak. Block’s own lesson is that support systems such as phone support need the same redundancy as core services.

Ably. A nearby, already-running second region kept a latency-sensitive service healthy. Ably needed only a DNS change to send new connections to us-east-2, at a worst-case cost of 12 ms at p50, while existing connections in us-east-1 stayed up. Ably moved because it could not add capacity in us-east-1 as traffic grew.

What did the teams that stayed up have in common?

The published accounts, plus AWS’s own resilience guidance, point to a common set of practices.

• They ran capacity in a second region before the incident.
• Their stateless services could be served from any region.
• Their data was replicated to the second region before the incident, so the recovery region had current data.
• Failover was a small, rehearsed step, such as a routing or DNS change or a Namespace failover, or fully automated. None of them depended on launching new capacity in us-east-1 during the incident.
• They kept us-east-1-only services out of the critical path, or found the ones that remained.
• FireHydrant, Block, and Ably cite game days or failover drills as the reason they could act quickly.
• They planned for partial impact: Block’s support systems and Temporal’s failover workflow showed where dependencies remained.

Is multi-AZ enough to survive a us-east-1 outage?

Availability zones protect against data center failures, power events, and localized network issues within a region. AWS designs each zone as an independent failure domain with separate power, cooling, and networking. Multi-AZ handles zone failures such as the May 7-8, 2026 single-zone event in the tracker.

A regional service is shared by every zone in the region. When the DynamoDB endpoint’s DNS record was emptied on October 19-20, 2025, services in every zone lost DynamoDB. EC2 instance launches and Network Load Balancer health checks were impaired across the region, so multi-AZ Auto Scaling groups could not reliably replace capacity. That event was regional and lasted 14h 32m. Surviving a regional failure requires a second region.

How can my company avoid being taken down by the next major outage?

For a deeper walkthrough, see how to keep your app online when an AWS region goes down and how to test failover without taking production down.

How do I reduce the blast radius of a cloud provider outage?

Cell-based architecture. Divide tenants or workloads across independent cells, each a full copy of the stack with no shared regional dependencies. A failure in one cell affects only the tenants assigned to it.

Staged deployments region by region. Roll changes out to one region at a time, with automated rollback when health checks fail, so a bad release stays in one region.

Static stability. Keep enough capacity running in each location to absorb a failure without launching instances or calling control plane APIs during the event, and use timeouts and circuit breakers so a degraded dependency does not cascade.

A second cloud provider for the most critical services. Running on two providers removes dependence on any one provider’s regional and global control planes. The operational cost is real, and it can be kept flat with one deployment workflow for both providers. See how to add a second cloud provider for failover without doubling your ops work and the cost of active-active vs active-passive.

Regulated industries face specific expectations here. See how financial services companies stay online during cloud outages.

How Control Plane fits

Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, and Azure, VM workloads on locations joined through Control Plane’s Managed Kubernetes, and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds.

A workload deployed to a GVC is active-active in every location of that GVC from one definition. Control Plane’s built-in global endpoint routes each request to the nearest healthy location, so failover is a routing change with no standby to start and no DNS change during the incident. Locations can be regions of one cloud, for example us-east-1 plus us-east-2 or us-west-2, or regions on AWS, GCP, and Azure together.

During the October 20, 2025 AWS us-east-1 outage, Control Plane customers with us-east-1 in their GVC had workloads active and running in healthy locations within 10 seconds, according to Control Plane’s internal incident telemetry. About 20% of Control Plane customers had us-east-1 in their GVC.

Platform routing moves stateless traffic. Stateful recovery depends on your replication design. Control Plane’s template catalog includes a Postgres Multi-Location template that promotes a replica automatically when it spans three or more locations, and a CockroachDB template that survives the loss of one of three locations.

Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security. On Control Plane-managed locations there is no Kubernetes cluster to operate. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. You pay for the CPU and memory a workload reserves, with no contracts or minimums. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.

FAQ

Short answers for teams planning for a us-east-1 or other regional outage.

How do companies stay online during AWS us-east-1 outages?

They run critical services in more than one region or cloud before the outage, replicate data to the second location, and fail over with a rehearsed routing change or automated health checks that do not depend on us-east-1 control planes. Temporal, FireHydrant, LoginRadius, Block, and Ably published accounts of how their multi-region designs kept core services running during the October 2025 outage.

How can my company avoid being taken down by the next major outage?

Map every dependency in your request path, run capacity in a second region or cloud, replicate data with measured lag, keep us-east-1-only control planes out of the failover path, and run game days that measure RTO and RPO. The 12-row checklist in this guide covers each step.

Which AWS region is the best backup for us-east-1?

us-west-2 (Oregon) is geographically distant and offers broad service availability. us-east-2 (Ohio) is closer, so latency to us-east-1 is lower; Ably moved new connections from us-east-1 to us-east-2 during the October 2025 outage with a worst-case latency impact of 12 ms at p50. Any AWS region still depends on us-east-1 for IAM, Route 53, and CloudFront control plane operations (Route 53 accelerated recovery covers public record changes if you turn it on in advance), and many customers fail over to the same nearby regions at once, so reserve capacity in the region you choose.

How much does it cost to run in a second region?

The main cost drivers are capacity headroom, duplicated data stores, and cross-region data transfer for replication. Active-passive adds the cost of the warm capacity you keep in the recovery region. In a two-region active-active setup, each region must absorb full peak traffic if the other fails, unless the surviving region can scale up during the event; three or more locations reduce the headroom each one needs. Control Plane customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure.

Do I need multi-cloud or is multi-region enough?

For most teams, multi-region is the right first step, because it covers regional failures like the October 2025 us-east-1 event. A second cloud also covers failures of one provider’s global control planes. On Control Plane, one workload definition runs active-active in every location of its GVC, whether those are several regions of one cloud or regions on AWS, GCP, and Azure together.

How do I test failover without taking production down?

Shift traffic away from the primary region in weighted steps during a low-traffic window, and at the same time block the recovery region’s outbound calls to primary-region endpoints with security groups, network ACLs, or AWS Fault Injection Service. The traffic shift tests routing, and the egress block exposes dependencies that still resolve to the primary region. Record time to full traffic shift, error rates, and replication lag at cutover. See how to test failover without taking production down.

What us-east-1 dependencies break multi-AZ designs?

Regional services shared by every zone. In October 2025 the trigger was the DNS record for the regional DynamoDB endpoint, and EC2 instance launches, Network Load Balancer health checks, Lambda, ECS, EKS, Fargate, and STS were impaired across the region. Global services with control planes in us-east-1, such as IAM, Route 53, and CloudFront, can also block changes during an event.

How often does us-east-1 have major outages?

Control Plane’s Cloud Outage Tracker records 9 of AWS’s 14 major outages since 2020 in us-east-1, a count not adjusted for region size. Three us-east-1 events since October 2025 each lasted more than 13 hours: the October 19-20 regional event, the October 28 event affecting EC2 launches and ECS, and the May 2026 single-zone event.

What is the difference between control plane and data plane during an AWS outage?

Control plane operations create, change, or delete resources, such as creating an IAM role or updating a Route 53 record. Data plane operations use existing resources, such as answering a DNS query or authorizing a signed request. During a us-east-1 event, control plane operations for global services can be impaired while data plane operations usually continue, so a failover plan should rely on data plane operations.

How does Control Plane keep workloads running during a regional outage?

A workload in a Control Plane GVC runs active-active in every location of that GVC, and the global endpoint routes each request to the nearest healthy location, so traffic shifts to the remaining locations automatically. See the October 2025 figures in How Control Plane fits.

To run your workloads active-active across regions and clouds, talk to the Control Plane team or sign up free.

Originally published on the Control Plane blog.