Part 2 of 3
In Part 1 of this series, you defined three business decisions that shape your disaster recovery (DR) commitment: the recovery boundary, the business-anchored target, and the dependency-reconciled commitment. Those decisions give you a target on paper. Testing tells you whether it holds under real conditions.
This post covers a progressive approach to DR validation. Each level of testing surfaces a different class of gap, and each gap feeds back into the three decisions from Part 1 so your commitment gets stronger with every cycle.
Why a single test isn’t enough
A scheduled DR exercise with advance notice proves that the failover mechanism works. That’s a necessary starting point. You can confirm that traffic shifts correctly, data remains consistent in the recovery Region, and the application functions as expected after the transition.
A scheduled exercise doesn’t test the assumptions that only surface under pressure. It doesn’t show whether teams who haven’t rehearsed together can coordinate, or whether everyone agrees on who declares the failover and under what conditions. It also can’t confirm that you provisioned and validated your recovery environment before you triggered the failover. These are the conditions where real recoveries stall, and they’re invisible to a test where every team knows it’s coming and watches closely.
In the same engagement from Part 1, several applications had DR instances in a stopped state. The failover automation assumed it could launch instances, create database replicas, and scale container fleets on demand during the exercise. That approach worked only if the team confirmed two things in advance: that the recovery AWS Region had capacity for the requested instance types, and that the account quotas could absorb the full production load. The team didn’t verify these assumptions in advance.
The architecture was correct. The team didn’t verify those assumptions in advance, and it didn’t pre-provision the recovery environment. To close this gap, the team needs to pre-provision the recovery environment and configure auto scaling groups, confirm that its account quotas match production capacity requirements, and remove any runbook step that assumes your team needs to create resources during the event.
Four levels of progressive validation
The AWS Resilience Lifecycle Framework treats testing as a practice you deepen over time. The four levels below are a maturity model you can apply yourself, ordered from lowest-cost awareness to highest-fidelity validation. We’ve drawn it from patterns across the DR exercises we’ve seen. Each surfaces a different class of gap.
Level 1: Tabletop exercise. A facilitated walkthrough of the recovery plan without touching production. The goal is shared awareness and ownership clarity.
Questions to ask during the tabletop:
- Who declares the failover, and under what conditions?
- For each step in the runbook, who owns it and what’s their backup?
- What’s the expected duration of each step, and what’s the total end-to-end timeline?
- Which steps depend on a specific person being available?
In the same case study from Part 1, the first tabletop surfaced three boundary gaps: the shared authentication platform wasn’t in the recovery scope, a networking account owned by a different team had no failover runbook, and the DNS update step listed a generic team alias with no named owner.
Level 2: Planned failover. A scheduled failover with advance notice. The goal is confirming the mechanism works under known conditions.
What to measure:
- Actual recovery time per application compared to the stated recovery time objective (RTO).
- Data consistency validation in the recovery Region to confirm the recovery point objective (RPO).
- Any step that requires manual intervention (potential automation candidate).
- Any dependency that recovered later than expected.
In the same case study, the planned failover revealed the DNS misconfiguration: the shared platform’s Amazon Route 53 records still resolved to the primary Region 45 minutes into the test. Most downstream applications appeared healthy but couldn’t serve users. Actual recovery took more than twice the 5-hour target.
A successful planned failover produces a measured gap (or confirmation) between the RTO target you defined and your actual recovery time. That number is the starting point for every subsequent conversation with the business owner about whether the target is realistic.
Level 3: Surprise drill. A failover triggered without advance notice to the responding team. The goal is testing execution readiness, not capability.
This level finds the gaps that preparation hides: the runbook that only its original author can interpret, the step that depends on a specific person’s tribal knowledge, and the escalation path that nobody has practiced. In the case study, a later surprise drill revealed that only one engineer understood how to validate database consistency in the recovery Region. With that person unavailable, the team lost 90 minutes locating and interpreting the procedure.
Level 4: Game day with fault injection. An exercise where you inject specific dependency failures using AWS Fault Injection Service, which ships purpose-built scenarios such as “AZ Availability: Power Interruption” and “Cross-Region: Connectivity”. The goal is testing resilience under conditions you didn’t fully script.
You discover the dependency you assumed would be healthy, the monitoring gap that delays detection, and the cascade that your tabletop didn’t model. This is the closest approximation to a real event without waiting for one to happen.
What each level reveals
The tabletop finds the step no team owns. The planned failover finds the misconfiguration. The surprise drill finds the coordination failure. The game day finds the unmodeled dependency. Each level surfaces a different class of gap.
Run only the calm, scheduled test and you validate the mechanism while leaving the conditions that cause real recovery failures untested. The point isn’t to skip levels. You build confidence at each level before increasing realism. But stopping at the planned failover, a pattern we often see once the mechanism works, leaves team, process, and dependency gaps invisible until an event exposes them.
Closing the loop
Each test produces specific findings that map directly to the three decisions from Part 1.
- A boundary gap (a shared service discovered mid-test that wasn’t in the recovery scope) feeds back into Decision 1.
- A recovery that exceeds the time the business can tolerate feeds back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
- A dependency that recovered slower than the target assumed feeds back into Decision 3 for a revised commitment that reflects what was actually measured.
In the same engagement, the first exercise surfaced all four patterns described across this series: a boundary gap (shared platform not in scope), a dependency gap (an upstream service whose recovery target capped the application’s own), a stability gap (resources not pre-provisioned), and a coordination gap (manual DNS failover). The team mapped each finding to a specific decision and a specific fix. Four months later, the second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time.
The value of progressive testing is not passing a single exercise. It’s building a practice where each cycle tightens the recovery commitment, and each finding has a clear path back to the decision it informs.
Where to start
If you haven’t run a tabletop yet, start there. Gather the owners of one critical business function and walk the recovery plan together. Ask: who declares the failover, what’s the first action, who owns each step, and what’s the expected duration of each. If the room disagrees with any answer, this indicates an opportunity to clarify and strengthen your recovery plan before touching production.
If you’ve passed a planned failover, consider whether a surprise drill would surface coordination gaps that preparation masks. In this engagement, the surprise drill cost the team 90 minutes that the planned failover never surfaced. Whatever level you’re at, document what each test reveals and feed it back into the three decisions. A posture that improves every test cycle is worth more than one that passes a single exercise and never evolves. In Part 3, we walk through how to structure a full DR readiness engagement: who to involve, what to produce at each step, and how to compress the timeline from months to weeks.
