Your service is running fine. Then a downstream dependency gets slow. Connections back up, goroutines pile up, your service's memory spikes, and suddenly you've got a full-service outage caused by a database that just had high latency for 30 seconds. This is a cascading failure — and it's entirely preventable with two patterns that belong in every production codebase: circuit breakers and bulkheads.

What circuit breakers actually do

A circuit breaker wraps calls to external dependencies. It tracks failure rates and, when things go south, opens the circuit — stopping all calls to the failing service for a cool-down period. This protects both your service (no more goroutine pile-up) and the downstream service (no more hammering a system that's struggling to recover).

The three states:

  • Closed: traffic flows normally
  • Open: traffic is blocked (fast-fail)
  • Half-open: a probe request is allowed to test recovery

Here's a minimal circuit breaker in Go:

package circuit import ( "errors" "sync" "time" ) type State int const ( StateClosed State = iota StateOpen StateHalfOpen ) var ErrCircuitOpen = errors.New("circuit breaker is open") type Breaker struct { mu sync.Mutex state State failureCount int threshold int lastFailure time.Time cooldown time.Duration } func New(threshold int, cooldown time.Duration) *Breaker { return &Breaker{threshold: threshold, cooldown: cooldown} } func (b *Breaker) Call(fn func() error) error { b.mu.Lock() switch b.state { case StateOpen: if time.Since(b.lastFailure) > b.cooldown { b.state = StateHalfOpen } else { b.mu.Unlock() return ErrCircuitOpen } } b.mu.Unlock() err := fn() b.mu.Lock() defer b.mu.Unlock() if err != nil { b.failureCount++ b.lastFailure = time.Now() if b.failureCount >= b.threshold { b.state = StateOpen } return err } b.failureCount = 0 b.state = StateClosed return nil } 

Usage:

breaker := circuit.New(5, 10*time.Second) err := breaker.Call(func() error { return callPaymentService() }) if errors.Is(err, circuit.ErrCircuitOpen) { return cachedPrice, nil } 

The threshold and cooldown values matter. For a payment service you might use 3 failures and a 30-second cooldown; for an internal cache 10 failures and 5 seconds. Tune these to your SLOs.

Bulkheads: isolating failure domains

The bulkhead pattern comes from ship design. A hull breach in one compartment doesn't sink the ship because compartments are sealed off. Applied to software: you give different clients or use cases their own resource pool.

Without bulkheads, a slow report-generation endpoint can consume all your worker goroutines and starve time-sensitive APIs. The fix is semaphore-based isolation:

package bulkhead import ( "context" "errors" ) var ErrBulkheadFull = errors.New("bulkhead capacity exceeded") type Bulkhead struct { sem chan struct{} } func New(capacity int) *Bulkhead { return &Bulkhead{sem: make(chan struct{}, capacity)} } func (bh *Bulkhead) Acquire(ctx context.Context) error { select { case bh.sem <- struct{}{}: return nil case <-ctx.Done(): return ctx.Err() default: return ErrBulkheadFull } } func (bh *Bulkhead) Release() { <-bh.sem } 

Wire this into your HTTP handlers per route group:

reportsBulkhead := bulkhead.New(10) // at most 10 concurrent report requests apiBulkhead := bulkhead.New(200) // 200 for time-sensitive API calls func reportsHandler(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) defer cancel() if err := reportsBulkhead.Acquire(ctx); err != nil { http.Error(w, "Service busy", http.StatusServiceUnavailable) return } defer reportsBulkhead.Release() generateReport(w, r) } 

Now a flood of report requests cannot degrade your API endpoints. When reportsBulkhead fills up, callers get a 503 immediately instead of waiting and consuming resources.

Combining both patterns

Circuit breakers and bulkheads solve different problems and belong together. The bulkhead limits concurrency within your service; the circuit breaker limits your exposure to upstream failures.

A realistic production wrapper:

type ResilientClient struct { breaker *circuit.Breaker bulkhead *bulkhead.Bulkhead } func (c *ResilientClient) Call(ctx context.Context, fn func() error) error { if err := c.bulkhead.Acquire(ctx); err != nil { return fmt.Errorf("bulkhead rejected: %w", err) } defer c.bulkhead.Release() return c.breaker.Call(fn) } 

This works for any external call: database queries, third-party APIs, internal microservices. If you need shared circuit state across replicas, back it with Redis — but start in-process and add complexity only when you hit actual problems.

Observability you actually need

Both patterns are silent by default. Add metrics:

func (b *Breaker) recordState() { state := "closed" if b.state == StateOpen { state = "open" circuitOpenTotal.WithLabelValues(b.name).Inc() } circuitState.WithLabelValues(b.name).Set(float64(b.state)) _ = state } 

The key signals to track:

  • circuit_open_total — how often the breaker is tripping
  • bulkhead_rejected_total — how often callers hit capacity
  • bulkhead_in_flight — current concurrency level

If circuit_open_total increases steadily, your downstream dependency has a reliability problem. If bulkhead_rejected_total spikes, you either need more capacity or a client is misbehaving. These signals surface incidents before your on-call gets paged.

For teams running production services, this kind of resilience hardening fits alongside security hardening. We publish free security and infrastructure checklists that cover both areas — worth a look if you're building production systems.

The takeaway

Circuit breakers and bulkheads are not optional extras — they are the difference between a contained incident and a full outage. Circuit breakers protect you from upstream degradation; bulkheads protect your upstream services from your own traffic. Together they give you controlled failure modes instead of unpredictable cascades.

Start simple: a 50-line semaphore bulkhead and a 70-line circuit breaker cover 90% of real-world cases. Add Redis-backed shared state only when your replica count makes in-process state unreliable. The hardest part is not the code — it's calibrating the thresholds to match your SLOs and actually acting on the metrics you expose.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.