TL;DR: "Self-healing infrastructure" gets thrown around loosely, usually meaning "it has a restart policy somewhere." The actual mechanism is two pillars working together: redundancy (so there's always a healthy replica to fail over to) and automated failover (so the system detects an unhealthy component and routes around it without a human). Neither one alone self-heals anything. A US payments processor's rebuild shows the mechanics end to end, plus the chaos-engineering discipline that proves it actually works before a real outage tests it for you.
A system with a Kubernetes liveness probe and a restart policy isn't self-healing in any meaningful sense, it's self-restarting, which only helps if the failure mode is "this one process crashed." Real resilience at the infrastructure level needs two things working together, and most teams only build one of them.
The two pillars that actually constitute "self-healing"
| Pillar | What it does | What breaks without it |
|---|---|---|
| Redundancy across zones/regions | Ensures a healthy replica exists somewhere when one fails | Failover has nowhere to fail over to |
| Automated failover | Detects an unhealthy component and reroutes without a human | Redundant capacity sits unused while an outage continues |
Redundancy without failover is just idle spare capacity, it exists, but nothing routes around the failure automatically, so you still page a human to flip the switch. Failover without redundancy has nothing to fail over to, the automation runs and finds no healthy target. The two have to be built together, which is why "we have multi-zone infrastructure" and "we're self-healing" are not the same claim, despite getting conflated constantly.
What this misses on its own: data and deploys
Redundancy and failover handle hardware and zone failure. They do nothing for the other two common failure modes: data loss (deleted records, ransomware, a bad migration) and bad deploys (a release that's technically "healthy" by every liveness check but wrong). That's a separate pillar, backups and disaster recovery, and it needs its own two numbers:
| Target | Question it answers |
|---|---|
| RTO (Recovery Time Objective) | How fast do we have to be back? |
| RPO (Recovery Point Objective) | How much data can we afford to lose? |
Set these per-system based on actual business need, not uniformly. A payments ledger earns a near-zero RPO (continuous replication, expensive). An internal analytics dashboard can tolerate an hourly backup. Applying payment-grade targets everywhere is an expensive way to avoid the harder prioritization conversation.
Proving it actually works: chaos engineering
A redundancy-plus-failover setup that's never been tested under real failure is a hypothesis, not a guarantee. Chaos engineering, deliberately killing a healthy instance, cutting off a zone, injecting latency, in a controlled window, is how you find out whether the failover logic actually triggers correctly before a 3am outage finds out for you. The uncomfortable finding teams usually get on their first chaos run: something in the failover path was silently broken, a stale health-check endpoint, a misconfigured load balancer rule, a runbook step nobody had actually executed. Finding that on a Tuesday afternoon costs nothing. Finding it during a real outage costs the outage.
Case study: from manual 4-hour recovery to automated 6-minute recovery
A US payments processor ran in a single region, so any zone-level problem caused a full outage, and recovery meant a person manually executing a runbook, which took hours while every minute down lost money. Six-month rebuild:
| Problem | What was built | Outcome |
|---|---|---|
| Single-region, no failover target | Multi-region, multi-zone redundancy | A healthy replica always exists somewhere |
| Manual, slow recovery | Automated failover and self-healing routing | Recovery: 4 hours → 6 minutes |
| Frequent downtime | Observability, alerting, scheduled DR drills | Uptime: 99.9% → 99.99% |
| Unproven DR plan | Tested backups and rehearsed failover, including chaos drills | Passed the DR audit on first attempt |
The 6-minute recovery number is the headline, but it's not one piece of technology, it's redundancy (pillar 1) and automated failover (pillar 2) built together, validated repeatedly through chaos drills until the failover path was something the team trusted rather than something they hoped worked.
FAQ
Is Kubernetes' default restart/health-check behavior enough for this?
It covers process-level failure within a node, useful but narrow. It doesn't cover zone or region failure, which needs redundancy across zones plus failover logic that operates above the single-cluster level.
How often should chaos engineering drills run?
Regularly enough that the team stays practiced and new changes get validated, monthly is a common cadence for mature setups; quarterly at minimum for teams just starting. The value compounds, each drill either confirms the system or surfaces a gap worth fixing immediately.
What's the fastest way to tell if our "self-healing" claim is real?
Kill a healthy instance in a non-production (or carefully scoped production) environment and watch whether traffic reroutes without anyone touching a keyboard. If it requires a runbook and a human, it's not self-healing yet, it's well-documented manual recovery.
Originally published on the Sherdil Cloud blog, the full piece (with the complete five-pillar breakdown and the cost-of-downtime analysis) is here. For the container/orchestration layer this builds on, see containerization done right; for the security side, cloud security best practices for enterprise.
About the author: Muhammad Usman is Head of DevOps at Sherdil Cloud, AWS DevOps Engineer Professional, Certified Kubernetes Administrator (CKA), and Alibaba Cloud Certified, building cloud and DevOps infrastructure for enterprises across Pakistan, the UAE, and the United States since 2014.
