How to Roll Back a Bad Change and Build Self-Healing Infrastructure: Safe Deploys, Smaller Blast Radius, Faster Recovery

Roll back a bad change by redeploying the last known-good version through the same pipeline that shipped the broken one, which only works if every deploy is versioned, health-gated, and reversible, and if schema and data changes were made backward compatible first. Reduce how often you need to roll back by shipping in small health-checked stages and running workloads across more than one failure domain, which is how Control Plane runs workloads: rolling updates where the previous version keeps serving, and active-active across every location in Control Plane’s Global Virtual Cloud (GVC), a set of locations a workload runs across.

Disclosure: Control Plane publishes this guide.

TL;DR

• Versioned, reversible deploys come first. Store every release as an immutable artifact. Rollback should take minutes.
• Roll back data last, and carefully. Destructive schema migrations and data changes cannot be undone with a traffic switch. Use expand-and-contract migrations so the old code still works while the new schema is live.
• Health-gated progressive delivery limits damage. Rolling updates, canary releases, blue/green switches, and feature flags each trade cost and complexity for faster rollback and smaller blast radius.
• Self-healing handles routine failures automatically. Health checks, automatic restarts, autoscaling, circuit breakers, and automatic traffic failover handle the failures that do not need a person.
• Blast radius shrinks with failure domains. Running across multiple availability zones, regions, or providers removes the single points of failure that turn a provider incident into your outage.
• Control Plane ties these together. Readiness-gated rolling updates, rollback through its MCP server, blue/green on domain routes, a tamper-proof audit trail, and active-active across every location in a GVC.

How do I roll back a bad AI-generated change in production?

Many outages start with a change: a bad deploy, a config push, a migration, or an AI-generated edit nobody fully reviewed. These steps apply whether a human or an AI agent wrote the change.

Step 1: Detect

Tie alerts to the signals that matter: error rate, request latency, CPU or memory saturation, and failed health checks. Synthetic monitors and real-user metrics catch what log-based alerts miss. The goal is to know something is wrong within one to two minutes of the change reaching production.

Step 2: Decide whether to roll back or fix forward

If the root cause is obvious and the fix is a one-line, well-tested change, fix forward. In every other case, roll back first, then investigate.

Step 3: Roll back

The mechanism depends on the change type:

• Application image: Redeploy the last known-good image digest through the same CI/CD pipeline or GitOps reconciliation loop.
• Config or environment variable: Revert the commit in the config repo or redeploy the previous config version.
• Feature flag: Toggle the flag off. This is the fastest rollback path because it requires no deploy.
• Infrastructure as code: Revert the commit and let the reconciliation loop or pipeline apply the previous state.
• Domain route (blue/green): Point the route back to the previous workload.

On Control Plane, an AI assistant can roll a workload back to its previous image (environment and config stay as they are) in one call through Control Plane’s MCP server, which makes the change, waits for the workload to report ready, and reports the result.

Step 4: Verify

Rollback is not complete until error rates, latency, and health probes return to baseline. Check business-critical user journeys, not just infrastructure metrics.

Step 5: Follow up

Preserve logs and traces. Run a blameless postmortem. Add a test that would have caught the failure before production.

Change types and how each is undone

The hard part: destructive schema and data changes

A column drop, a table rename, or a data backfill that overwrites values cannot be undone by redeploying an image. The solution is expand-and-contract (parallel change) migrations:

1. Expand: Add the new column or table alongside the old one. Deploy code that writes to both and still reads the old one.
2. Migrate: Backfill existing data into the new structure.
3. Switch reads: Deploy code that reads the new structure.
4. Contract: Drop the old column only after rollback to the old version is no longer needed.

This pattern keeps the previous application version compatible with the current schema at every step.

Guardrails for AI-generated changes

AI coding assistants and infrastructure agents accelerate development, but they also produce changes that humans may not fully read before merging. Add these guardrails:

• Require human review for any change that touches a production path.
• Keep diffs small. A 2,000-line AI-generated PR is effectively unreviewed.
• Run the full test suite in CI before merge.
• Keep an audit trail that shows whether a person or an agent made a change, and who approved it.

On Control Plane, a tamper-proof audit trail records who changed which workload and when, including service accounts and AI agents using the MCP server (cpln workload audit), and the MCP server offers a read-only profile for agents that should observe without changing anything. For more on agent permissions, see how to let AI agents touch production infrastructure safely.

Deploy patterns that limit damage

The right pattern depends on how fast you need to roll back, your infrastructure budget, and how mature your observability is.

Rolling updates

Replace instances of the old version with the new version incrementally. If new replicas fail their readiness checks, the rollout stalls while old replicas keep serving. Kubernetes does not roll back automatically (you run kubectl rollout undo), and tools like Argo Rollouts add automated rollback. Rolling updates are also the default on Control Plane, where rollout options such as minReadySeconds, maxUnavailableReplicas, and maxSurgeReplicas control how many replicas turn over at once.

Blue/green deployments

Run two full environments. Route traffic to one while deploying to the other, cut over when the new environment is healthy, and roll back by switching the route back. On Control Plane, blue/green uses domain routes, and traffic returns to the previous workload within seconds of pointing the route back.

Canary releases

Send a small share of traffic to the new version, watch error rates and latency, and promote or roll back based on the results. Argo Rollouts and Flagger automate canary analysis in Kubernetes.

Feature flags

Decouple deployment from release. Ship code dark, then enable it for specific users or percentages, and roll back by toggling the flag. OpenFeature provides a vendor-neutral API for flag evaluation.

The hyperscalers offer managed versions of these patterns. AWS CodeDeploy supports in-place (rolling) deployments for EC2 and on-premises servers, blue/green deployments for EC2, and canary, linear, or all-at-once traffic shifting for Lambda and ECS. Google Cloud Deploy provides managed continuous delivery to GKE, Cloud Run, and GKE attached clusters, with approvals, canary deployments, and rollback. Azure Pipelines supports multi-stage pipelines with approvals and checks, and deployment strategies whose on-failure hooks can run rollback steps.

Which CI/CD and IaC solutions offer the best disaster recovery and self-healing patterns for cloud native applications?

The patterns matter more than the vendor. Together, they recover from failed deploys, infrastructure drift, and the loss of an environment.

GitOps reconciliation

Argo CD and Flux continuously compare a cluster with the desired state declared in Git. When someone edits a resource by hand or a deploy goes wrong, reconciliation restores the declared state (Flux does this by default; Argo CD requires automated sync with self-heal enabled). Rolling back means reverting a commit.

Immutable images and infrastructure as code

Reference every release by its content-addressable digest, not a mutable tag like latest, so a rollback redeploys exactly the same binary. Terraform, OpenTofu, and Pulumi support remote state backends, most with state locking, to prevent concurrent changes. Back up state files, because without valid state these tools cannot tell what exists. Control Plane ships its own Terraform and Pulumi providers, so workload definitions can go through the same review process as the rest of your infrastructure, and workloads can be promoted from a staging GVC to a production GVC.

GitOps restores compute and configuration. It does not restore databases, third-party API state, or DNS records managed outside the reconciliation loop. Those need their own backup and recovery plan.

Self-healing infrastructure

Self-healing means the system detects an unhealthy condition and takes a bounded corrective action without waiting for a human. Each mechanism handles a specific failure mode.

Health checks

• Liveness probes answer “is this process stuck?” If a liveness check fails repeatedly, the kubelet or platform restarts the container.
• Readiness probes answer “can this replica handle traffic?” A replica that fails readiness stops receiving traffic but is not restarted.
• Startup probes give slow-starting applications time to initialize before liveness and readiness checks begin.

On Control Plane, readiness and liveness probes can be HTTP, gRPC, TCP, or a command, and they decide when a replica takes traffic and when it is restarted.

Restarts, rescheduling, and autoscaling

When a container crashes or a node fails, the orchestrator restarts the container or reschedules it onto a healthy node. Horizontal autoscaling adds replicas as demand rises, and right-sizing adjusts CPU and memory to actual usage. On Control Plane, built-in autoscaling and Capacity AI, Control Plane’s right-sizing of CPU and memory based on historical usage, handle both.

Circuit breakers, timeouts, and retries with backoff

When a downstream dependency fails, circuit breakers stop sending it traffic, which prevents cascading failures. Timeouts bound how long a request waits. Retries with exponential backoff and jitter recover from transient errors without overwhelming the failing service. Retry only idempotent operations, or use idempotency keys, to avoid duplicate side effects.

Automatic traffic failover between locations

Restarting a container fixes a process-level failure on one node. Failing traffic over to another location fixes a zone-level or region-level failure, and that requires workloads already running in more than one location plus a global load balancer that detects location health. On Control Plane, a workload deployed to a GVC runs active-active in every location of that GVC, and a global endpoint routes each request to the nearest healthy location. Priority routing tiers (a lower-priority tier receives traffic only when every location in the higher-priority tier is unavailable) and latency offsets let you shape where traffic goes.

How do I reduce the blast radius of a cloud provider outage?

Blast radius applies to your own changes and to your provider’s failures.

Staged rollouts by location

Deploy to one location first, watch it, then promote to the rest. A bad change then reaches only a fraction of your traffic.

Cell-based architecture

Partition users or tenants into independent cells, each with its own compute, data, and dependencies, so a failure in one cell does not reach the others.

Availability zones, regions, and providers

• Multiple availability zones protect against single-datacenter failures. Many regional managed services, such as S3 and DynamoDB, replicate across zones automatically, but EC2 instances, EBS volumes, and single-AZ RDS databases are zonal unless you configure Multi-AZ.
• Multiple regions protect against region-wide outages, which are rarer but have much higher impact.
• Multiple providers protect against provider-wide failures and reduce dependence on any single vendor.

Each step adds complexity and cost; see active-active vs active-passive cost for the numbers.

Hidden single-region dependencies

Your compute may be multi-region, but if your DNS provider, identity provider, CI/CD system, secrets manager, container registry, or a provider management API is pinned to one region, that region is still a single point of failure. Audit every dependency in the request path.

Rate limiting, load shedding, and kill switches

When a failure reduces capacity, rate limiting and load shedding protect the remaining healthy capacity; shed low-priority traffic first. A kill switch is a pre-built way to disable a feature, a dependency call, or a traffic path instantly, and it should be tested regularly.

For the region-failure playbook, see how to keep your app online when an AWS region goes down. Control Plane’s cloud outage tracker records how often these events happen.

What is the best cloud architecture model that delivers the highest level of resilience and availability?

The answer depends on how much downtime and data loss your business can tolerate, so start from your recovery targets.

How do I design cloud resilience and failover to meet a specific RTO and RPO?

Recovery time objective (RTO) is how fast you need to be back. Recovery point objective (RPO) is how much data you can afford to lose. Your data replication strategy sets RPO; compute architecture does not.

Near-zero RPO requires synchronous or near-synchronous data replication. Active-active gives the lowest RTO because failover takes only as long as health checks need to mark a location unhealthy and reroute traffic, typically seconds, with no standby to start or scale up. A multi-region compute layer paired with a single-region database does not give you multi-region resilience, so databases need their own replication and rollback design.

How Control Plane approaches change safety and recovery

Control Plane is the platform for running unbreakable apps and agents. It runs serverless, standard, stateful, and cron workloads on AWS, GCP, and Azure, on Kubernetes clusters you already run, and on your own machines turned into managed locations, including on premises. Many teams run it in one cloud as well.

Every workload update is a rolling update. The previous version keeps serving while the new one starts, and the update completes only when the new version reports ready. If the new version never becomes ready, the rollout stops and the previous version keeps handling traffic until you fix the workload or roll it back.

The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. Those locations can be two regions of one cloud or locations on different providers.

During the October 20, 2025 AWS us-east-1 outage, which lasted about 15 hours, Control Plane customers with us-east-1 in their GVC had no downtime, and their workloads were “active and running in healthy locations within 10 seconds,” according to Control Plane internal incident telemetry.

Control Plane handles Day 2 operations for you, including autoscaling, Capacity AI right-sizing, built-in observability, and security, with no Kubernetes cluster you have to operate. Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.

For the broader checklist, see how to prevent website downtime.

FAQ

These answers cover the questions teams ask most about rollback, self-healing, and blast radius.

How do I roll back a bad deployment quickly?

Redeploy the last known-good image or config through the same pipeline that shipped the broken version. That requires every release to be an immutable, versioned artifact and a pipeline that can deploy a specific version, not just the latest.

What is the difference between rolling back and fixing forward?

Rolling back restores the previous version. Fixing forward ships a new version that corrects the problem. Roll back when the root cause is unclear or the fix is not trivial, and fix forward only when the fix is small, tested, and faster than a rollback.

How do I roll back a database migration?

Use expand-and-contract migrations. Add the new columns or tables alongside the old ones, deploy code that writes to both and reads the old one, backfill, switch reads to the new structure, then drop the old structure once rollback is no longer needed. Destructive changes like column drops or renames cannot be reversed by redeploying application code.

Which deploy pattern has the fastest rollback?

Feature flags are the fastest because toggling a flag is instant and needs no deploy. Blue/green and canary rollbacks take seconds because only traffic moves. Rolling updates take longer because replicas must be replaced.

What is self-healing infrastructure?

Self-healing infrastructure detects unhealthy conditions and takes bounded corrective actions: restarting a failed container, removing an unhealthy replica from traffic, scaling up to meet demand, or failing traffic over to a healthy location. It does not replace human judgment for complex or ambiguous failures.

How do I reduce the blast radius of a cloud provider outage?

Run workloads across more than one availability zone, region, or provider, and remove hidden single-region dependencies like DNS, identity, CI/CD, secrets, and container registries. On Control Plane, workloads run active-active across every location in a GVC, and traffic fails over to healthy locations automatically.

What RTO and RPO can I achieve with multi-region active-active?

Multi-region active-active delivers an RTO measured in seconds. RPO near zero requires synchronous replication, which adds cross-region write latency; asynchronous replication gives an RPO equal to the replication lag, usually seconds.

How do I make AI-generated changes safer in production?

Require human review on production paths, keep diffs small, run tests before merge, deploy in health-gated stages, and keep an audit trail of what each agent changed. On Control Plane, agents using the MCP server act through the same API as everyone else, so every change lands in the audit trail, and a read-only profile is available for agents that should only observe.

To run workloads active-active with readiness-gated rollouts, talk to the Control Plane team or sign up free.

Originally published on the Control Plane blog.