A dependency doesn't have to go completely offline to cause trouble. Sometimes it just gets slow enough that requests pile up, workers stay busy, and retries add even more traffic to a service already struggling. A circuit breaker gives that dependency breathing room—but only if the timeouts and retries around it are configured with the same care.

The Problem

Working with multiple LLM providers (OpenAI, Anthropic, Google) presents several challenges. Each provider has different failure modes, rate limits, and latency characteristics. Without proper handling, a slow or failing provider can bring down an entire pipeline.

The naive approach is to add timeouts and retries. But this creates new problems:

  • Retries during an outage amplify the problem (retry storms)
  • Long timeouts block resources and cascade failures
  • Users wait for failures we already know will happen

What is a Circuit Breaker?

The circuit breaker pattern, popularized by Michael Nygard in "Release It!", works like an electrical circuit breaker. When failures exceed a threshold, the circuit "opens" and immediately fails requests without attempting them.

Three states:

  1. Closed - Normal operation. Requests pass through.
  2. Open - Failure threshold exceeded. Requests fail immediately.
  3. Half-Open - Testing if the service recovered. Let a single probe request through.

The half-open state is where most implementations go wrong. If the dependency is just coming back, the worst thing you can do is send it the full load again. One probe is enough to learn whether it recovered.

Implementation

Here's a simplified circuit breaker. It is per-process: the lock protects threads in one process, and nothing here coordinates between processes (more on that below).

import time from datetime import timedelta from enum import Enum from threading import Lock class CircuitState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" class CircuitBreaker: def __init__( self, failure_threshold: int = 5, recovery_timeout: timedelta = timedelta(seconds=30), probe_lease: timedelta = timedelta(seconds=10), ): self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.probe_lease = probe_lease self.state = CircuitState.CLOSED self.failure_count = 0 self.opened_at = 0.0 self.probe_deadline = 0.0 # when the current probe's claim expires self.lock = Lock() def can_execute(self) -> bool: with self.lock: now = time.monotonic() if self.state == CircuitState.CLOSED: return True if self.state == CircuitState.OPEN: if now - self.opened_at < self.recovery_timeout.total_seconds(): return False # Timeout over: this caller claims the probe. self.state = CircuitState.HALF_OPEN self.probe_deadline = now + self.probe_lease.total_seconds() return True # Half-open: one probe is already out. Everyone else fails fast, # unless the probe's lease expired (the prober never reported back). if now >= self.probe_deadline: self.probe_deadline = now + self.probe_lease.total_seconds() return True return False def record_success(self): with self.lock: if self.state == CircuitState.HALF_OPEN: self.state = CircuitState.CLOSED self.failure_count = 0 def record_failure(self): with self.lock: if self.state == CircuitState.OPEN: return # a late result from a call that started before we opened if self.state == CircuitState.HALF_OPEN: self._open() # the probe failed: back to open, restart the timer return self.failure_count += 1 if self.failure_count >= self.failure_threshold: self._open() def _open(self): self.state = CircuitState.OPEN self.opened_at = time.monotonic()

Two things changed from the naive version, and both matter:

  • Only one probe passes. The caller that moves the circuit from open to half-open claims the probe, and every other caller is refused until that probe reports. An implementation that lets "a few" calls through in half-open sends several requests at a service that may still be fragile. (An earlier version of this example did exactly that: the call that flipped the state, plus up to three more.)
  • A probe has a lease. If the probing caller crashes or forgets to call record_success or record_failure, the circuit would otherwise sit in half-open forever. The lease lets another caller try after a bounded time.

Concurrency across processes

The lock above is a thread lock. If you run twenty workers, you have twenty independent breakers, and each one sends its own probe: twenty probes, not one. For many services that is acceptable, and per-process breakers are common because they are simple and add no dependency. But say it plainly: this example gives you one probe per process.

If you need one probe for the whole fleet, the claim must be atomic in shared state. A short-lived lease in Redis is one way:

# Returns True for exactly one caller until the lease expires. claimed = redis.set(f"breaker:{service}:probe", worker_id, nx=True, ex=10)

SET ... NX EX succeeds for only one caller, and the expiry is the lease: if the prober dies, the key vanishes and someone else can claim it. The costs are real, though. Every call now depends on Redis being reachable, so decide what the breaker does when Redis is down (usually: fall back to the local breaker). A database row with UPDATE ... WHERE probe_claimed_until < now() works the same way.

Timeouts and Retries

A breaker can only count failures that the caller actually gives up on. If a call hangs for two minutes, it is neither a success nor a failure for that whole time, and the threshold is never reached while workers pile up behind it. So:

  • Put a timeout on every call under the breaker, and make it shorter than whatever deadline the user or upstream caller is waiting on. Count a timeout as a failure.
  • Decide where retries sit. If retries wrap the breaker, each attempt is counted and an open circuit stops further attempts. If retries are inside the breaker call, the breaker sees one slow failure instead of several quick ones, and an attempt can carry on while the breaker is trying to open. Retry outside, and don't retry when the circuit is open.
  • Remember that layers multiply. If a service, its client library and the HTTP client each retry three times, one request can become 27 calls to a dependency that is already struggling.

Retries are what turn a partial outage into a retry storm. Three habits keep them safe:

  1. Cap attempts at a small number.
  2. Back off with jitter. Wait exponentially longer between attempts, and randomize the wait, so a thousand clients don't all come back on the same second.
  3. Use a retry budget. Allow retries only up to a fraction of normal traffic (say, 10 percent, as an illustrative number) so that during an outage the extra load stays bounded instead of doubling or tripling.

Retrying also requires the operation to be safe to repeat. For a write, send an idempotency key so a retry after an ambiguous timeout doesn't do the work twice.

Key Decisions

Failure threshold: The example counts consecutive failures, and a success resets the count. A rolling window of recent calls is better when traffic is mixed, because it can tell "5 failures in a row" from "5 failures out of 10,000". Too low a threshold and you'll open on transient errors. Too high and you'll let too many requests fail first.

Recovery timeout: 30 seconds is a reasonable starting point for many APIs, not a rule. For rate-limited services, we use longer timeouts based on the rate limit reset time.

What counts as failure: Not all errors should trip the breaker. In the example, we count:

  • Timeouts, including connection timeouts
  • Connection failures
  • 5xx errors

Client errors (4xx) are the caller's problem, not a service failure. A rate-limit response (429) is a signal to slow down, so handle it with backoff and the provider's retry-after hint instead of counting it toward opening the circuit.

Fallback Strategies

An open circuit isn't useful if you just return an error. We use several fallback approaches:

  1. Provider failover: If OpenAI's circuit opens, route to Anthropic
  2. Cached responses: Return stale data if appropriate
  3. Graceful degradation: Offer reduced functionality
  4. Queue for retry: Persist the request and process later

Monitoring

Circuit state changes are important signals. We track:

  • Circuit open events (alerts)
  • Time spent in each state
  • Fallback success rates
  • False positive rate (opened when service was healthy)
  • Retry rate, as a share of total calls. A rising retry rate often shows up before the breaker opens.

Lessons Learned

Start conservative: It's easier to relax thresholds than to explain cascading failures.

Test the open state: Your fallback logic needs as much testing as the happy path. Test half-open with concurrent callers too, and check that only one request goes out.

Different breakers for different operations: Read and write operations to the same service might need different thresholds.

Document the behavior: Teams need to know what happens when a circuit opens.

Circuit breakers are one piece of a resilience strategy. Combined with timeouts, bulkheads, and proper retry policies, they help systems degrade gracefully instead of failing all at once.

Sources and further reading

The primary material behind the pattern, and what to read once the basics land.

  • Circuit breaker by Martin Fowler. The canonical description of closed, open and half-open, and what each state is for. The idea comes from Michael Nygard's Release It!, which is worth reading whole.
  • Addressing cascading failures from the Google SRE book. What actually happens when a dependency degrades and every caller retries into it. This is the failure a breaker exists to stop.
  • Handling overload, same book, for load shedding and graceful degradation, which are the tools a breaker hands off to once it opens.
  • Timeouts, retries and backoff with jitter in the Amazon Builders' Library. A breaker without timeouts underneath it never trips, because nothing ever fails fast enough to count.
  • Exponential backoff and jitter from the AWS Architecture Blog, for what retries should look like when many clients recover at once.

Continue learning

If the three states now make sense, the next thing to understand is timeouts, because a breaker can only count failures that the caller actually gives up on. After that, bulkheads, which stop one failing dependency from consuming the thread pool everything else needs. Then load shedding, which is what you do when the problem is your own service rather than a dependency.

FAQ

Where should breaker state live?
Per-process is the simplest and is often enough: no extra dependency, and each worker protects itself. The cost is that every worker discovers the outage on its own, and sends its own half-open probe. Shared state in Redis or a database gives you one view and one probe for the fleet, but adds a dependency you must handle when it fails. If you share state, make the probe claim atomic.

What failure rate should open the breaker?
Start from what a healthy dependency looks like, not from a round number. If it normally errors on 1% of calls, a 50% threshold over a meaningful window is a real signal, while 5% is noise that can trip during a routine blip.

Should a breaker cover all errors?
No. Count the ones that mean the dependency is unhealthy: timeouts, connection failures, 5xx. A 400 means your request was wrong and will fail identically on every retry, and a 429 is a rate limit that backoff handles better.

Should I retry inside the breaker?
Prefer retrying outside it, with a capped number of attempts, jittered backoff and a retry budget, and stop retrying as soon as the circuit is open.

Originally published at www.internalswithzee.com/blog/circuit-breakers.