Circuit breaker
A resiliency pattern that stops calling a failing downstream after a threshold, letting it recover instead of hammering it. It has closed, open, and half-open states. While open, calls fail fast or return fallbacks.
- Outbound calls to third parties or cross-service deps that can fail.
- Preventing retry amplification during partial outages.
- Idempotent background jobs where you can just retry later.
- Calls where failing fast is worse than waiting (e.g. critical writes).
Definition
Wraps a call with a state machine: Closed calls normally and counts failures; Open short-circuits all calls for a cooldown; Half-Open lets a probe through to test recovery. Tripping uses a rolling failure count or rate over a window.
Why it exists
When a dependency degrades, cascading retries and threads blocked on timeouts can take down the caller too. A breaker fails fast, frees resources, and gives the downstream room to recover.
How it works
Track success/failure in a sliding window. If failure rate exceeds threshold (e.g. 50% of last 100 calls), open the circuit. After a sleep window, allow one trial call; success closes, failure re-opens. Often paired with timeouts and bulkheads.
Scaling limits
Per-dependency state is cheap. The hard part is tuning thresholds so it trips on real outages, not transient blips, across varying traffic.
Cost shape
Negligible compute. The cost is operational: mis-tuned breakers cause false trips that drop traffic.
Alternatives
- Timeout — Bounds a single call but doesn't stop repeated failures.
- Bulkhead — Isolates failures by limiting concurrency, not by tripping.
Failure modes
- Tripping too early on a blip and shedding good traffic.
- Never half-opening, so recovery is delayed.