Circuit Breaker Pattern Implementation — Resilience in Distributed Systems
In this tutorial, you'll learn about Circuit Breaker Pattern Implementation. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
The Circuit Breaker Pattern prevents cascading failures in Distributed Systems by detecting when a downstream service is failing and temporarily stopping requests to it, giving it time to recover.
What You'll Learn
By the end of this tutorial, you will implement circuit breakers in Python and Node.js, configure failure thresholds and timeouts, integrate with HTTP clients and database connections, and combine circuit breakers with retry and fallback patterns.
Why It Matters
When one service fails, every other service that depends on it also fails if they keep making requests. This cascading failure can bring down an entire system. Doda Browser's backend uses circuit breakers to isolate failures in the file scanning service, preventing scan timeouts from affecting the upload API.
Real-World Use
A payment service calls an external fraud detection API. If the fraud API starts returning 500 errors, the payment service's circuit breaker opens. Subsequent payment requests skip the fraud check (fallback) instead of timing out. After 30 seconds, the circuit allows a test request. If it succeeds, the circuit closes and normal flow resumes.
Circuit Breaker States
stateDiagram-v2
[*] --> Closed
Closed --> Open: Failure threshold exceeded
Open --> HalfOpen: Timeout elapsed (reset_timeout)
HalfOpen --> Closed: Probe request succeeds
HalfOpen --> Open: Probe request fails
Open --> Closed: Manual reset
note right of Closed
Requests pass through
Failure counter increments
end note
note right of Open
Requests Fail Fast
No downstream calls
end note
note right of HalfOpen
Limited probe requests
Test if service recovered
end note
The circuit starts in Closed State. When failures exceed the threshold, it opens and all requests Fail Fast. After a timeout, it transitions to Half-Open and allows probe requests. Success closes the circuit; failure reopens it.
Python: Circuit Breaker Implementation
# circuit_breaker.py
# Generic circuit breaker implementation
import time
import threading
from enum import Enum
from typing import Callable, Optional, Any
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
"""Circuit breaker for resilient external service calls."""
def __init__(
self,
name: str,
failure_threshold: int = 5,
recovery_timeout: float = 30.0,
half_open_max_calls: int = 3,
):
self.name = name
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max_calls = half_open_max_calls
self.State = CircuitState.CLOSED
self.failure_count = 0
self.last_failure_time = 0.0
self.half_open_calls = 0
self.lock = threading.Lock()
def call(self, func: Callable, fallback: Optional[Callable] = None, *args, **kwargs) -> Any:
"""Execute the function with circuit breaker protection."""
if self.State == CircuitState.OPEN:
if self._should_attempt_recovery():
self._transition_to_half_open()
else:
return self._handle_open(fallback, *args, **kwargs)
try:
result = func(*args, **kwargs)
self._on_success()
return result
except Exception as e:
self._on_failure()
if fallback:
return fallback(*args, **kwargs)
raise
def _should_attempt_recovery(self) -> bool:
"""Check if recovery timeout has elapsed."""
return (time.monotonic() - self.last_failure_time) >= self.recovery_timeout
def _transition_to_half_open(self):
"""Transition from Open to Half-Open State."""
with self.lock:
if self.State == CircuitState.OPEN:
print(f"Circuit [{self.name}] transitioning to HALF_OPEN")
self.State = CircuitState.HALF_OPEN
self.half_open_calls = 0
def _on_success(self):
"""Handle successful execution."""
with self.lock:
if self.State == CircuitState.HALF_OPEN:
self.half_open_calls += 1
if self.half_open_calls >= self.half_open_max_calls:
print(f"Circuit [{self.name}] closing (recovered)")
self.State = CircuitState.CLOSED
self.failure_count = 0
self.half_open_calls = 0
elif self.State == CircuitState.CLOSED:
self.failure_count = 0 # Reset on consecutive success
def _on_failure(self):
"""Handle failed execution."""
with self.lock:
self.failure_count += 1
self.last_failure_time = time.monotonic()
if self.State == CircuitState.HALF_OPEN:
print(f"Circuit [{self.name}] reopening (probe failed)")
self.State = CircuitState.OPEN
elif self.failure_count >= self.failure_threshold and self.State == CircuitState.CLOSED:
print(f"Circuit [{self.name}] opening ({self.failure_count} failures)")
self.State = CircuitState.OPEN
def _handle_open(self, fallback: Optional[Callable], *args, **kwargs) -> Any:
"""Handle request while circuit is open."""
print(f"Circuit [{self.name}] OPEN - failing fast")
if fallback:
return fallback(*args, **kwargs)
raise CircuitBreakerOpenError(f"Circuit [{self.name}] is open")
def reset(self):
"""Manually reset the circuit to closed State."""
with self.lock:
self.State = CircuitState.CLOSED
self.failure_count = 0
self.half_open_calls = 0
print(f"Circuit [{self.name}] manually reset to CLOSED")
class CircuitBreakerOpenError(Exception):
"""Raised when a circuit is open and no fallback is provided."""
pass
# Usage
breaker = CircuitBreaker(name="payment-API", failure_threshold=3, recovery_timeout=30)
def call_payment_API(order_id):
"""Simulated external API call."""
import random
if random.random() < 0.7: # 70% failure rate
raise ConnectionError("Payment API timeout")
return {"status": "success", "order_id": order_id}
def payment_fallback(order_id):
"""Fallback when circuit is open."""
return {"status": "fallback", "order_id": order_id, "note": "Queued for retry"}
for i in range(10):
result = breaker.call(call_payment_API, payment_fallback, f"ORD-{i}")
print(f"Request {i}: {result}")
Node.js: Circuit Breaker with Axios
// circuit-breaker.js
// Circuit breaker with Axios HTTP client
const Axios = require('Axios');
class CircuitBreaker {
constructor(options = {}) {
this.failureThreshold = options.failureThreshold || 5;
this.recoveryTimeout = options.recoveryTimeout || 30000;
this.halfOpenMaxRequests = options.halfOpenMaxRequests || 3;
this.timeout = options.timeout || 5000;
this.State = 'CLOSED';
this.failureCount = 0;
this.lastFailureTime = null;
this.halfOpenRequests = 0;
}
async call(requestConfig) {
if (this.State === 'OPEN') {
if (this.shouldAttemptRecovery()) {
this.transitionTo('HALF_OPEN');
} else {
return this.failFast(requestConfig);
}
}
try {
const response = await Axios({
...requestConfig,
timeout: this.timeout,
});
this.onSuccess();
return response.data;
} catch (error) {
this.onFailure(error);
throw error;
}
}
shouldAttemptRecovery() {
return Date.now() - this.lastFailureTime >= this.recoveryTimeout;
}
transitionTo(newState) {
console.log(`Circuit [${requestConfig?.URL}] transitioning to ${newState}`);
this.State = newState;
if (newState === 'HALF_OPEN') {
this.halfOpenRequests = 0;
}
}
onSuccess() {
if (this.State === 'HALF_OPEN') {
this.halfOpenRequests++;
if (this.halfOpenRequests >= this.halfOpenMaxRequests) {
console.log('Circuit closed - service recovered');
this.State = 'CLOSED';
this.failureCount = 0;
this.halfOpenRequests = 0;
}
} else if (this.State === 'CLOSED') {
this.failureCount = 0;
}
}
onFailure(error) {
this.failureCount++;
this.lastFailureTime = Date.now();
if (this.State === 'HALF_OPEN') {
console.log('Circuit reopened - probe failed');
this.State = 'OPEN';
} else if (
this.State === 'CLOSED' &&
this.failureCount >= this.failureThreshold
) {
console.log(`Circuit opened after ${this.failureCount} failures`);
this.State = 'OPEN';
}
}
failFast(requestConfig) {
const error = new Error(`Circuit breaker OPEN for ${requestConfig?.URL}`);
error.code = 'CIRCUIT_OPEN';
error.status = 503;
throw error;
}
getState() {
return {
name: this.name,
State: this.State,
failureCount: this.failureCount,
lastFailure: this.lastFailureTime
? new Date(this.lastFailureTime).toISOString()
: null,
recoveryIn: this.lastFailureTime
? Math.max(0, this.recoveryTimeout - (Date.now() - this.lastFailureTime))
: 0,
};
}
}
// Usage with Express
const Express = require('Express');
const app = Express();
const paymentCircuit = new CircuitBreaker({
failureThreshold: 3,
recoveryTimeout: 30000,
});
app.post('/checkout', async (req, res) => {
try {
const result = await paymentCircuit.call({
method: 'POST',
URL: 'HTTPS://payment-API.example.com/charge',
data: req.body,
});
res.JSON(result);
} catch (error) {
if (error.code === 'CIRCUIT_OPEN') {
// Use cached response or Queue for later
return res.status(503).JSON({
error: 'Payment service unavailable',
retryAfter: Math.ceil(paymentCircuit.getState().recoveryIn / 1000),
});
}
res.status(502).JSON({ error: 'Payment failed' });
}
});
// Circuit State endpoint
app.get('/circuits', (req, res) => {
res.JSON({
payment: paymentCircuit.getState(),
});
});
Common Errors
1. Setting the Failure Threshold Too Low
A threshold of 1-2 failures causes the circuit to open during transient blips (network jitter, brief timeouts). Set the threshold to 5-10 failures within a rolling window to distinguish between transient errors and sustained failures.
2. Not Implementing Half-Open State
Without half-open, a circuit opens and stays open permanently until manual intervention. The half-open State allows automatic recovery by sending probe requests. Without it, every failure requires manual reset.
3. Mixing Timeouts with Failures
All circuit breakers treat timeouts as failures. If your timeout is too short (e.g., 100ms for a database query), normal slow queries trigger the circuit. Set realistic timeouts based on P99 latency measurements.
4. No Fallback for Open Circuit
When the circuit opens, requests fail with exceptions. Without a fallback, the caller must handle the exception. Implement fallbacks: return cached data, Queue the request for later, or use a degraded response.
5. Circuit Breaker for Local Resources
Circuit breakers are designed for remote service calls (HTTP, RPC). Using them for local function calls or in-memory caches adds unnecessary complexity. Use circuit breakers at service boundaries only.
6. Not Monitoring Circuit State
An open circuit silently degrades functionality. Monitor circuit State changes, failure counts, and recovery attempts. Alert when a circuit opens, especially for critical dependencies.
Practice Questions
1. What are the three states of a circuit breaker?
Closed (normal operation, requests pass through), Open (requests Fail Fast without calling the downstream service), and Half-Open (limited probe requests to test recovery).
2. How does the circuit breaker prevent cascading failures?
When a downstream service fails, the circuit opens and subsequent requests fail immediately without waiting for timeouts. This prevents upstream services from accumulating connections and threads waiting for responses, which prevents resource exhaustion.
3. What is the difference between circuit breaker and retry?
Retry repeats the same failed request, expecting it to succeed. Circuit breaker stops all requests to a failing service to give it time to recover. They complement each other: use retry for transient errors, use circuit breaker for sustained failures.
4. How do you determine the recovery timeout?
BASE it on the expected recovery time of the downstream service. A database failover takes 10-30 seconds. A service restart takes 30-60 seconds. Monitor historical recovery times and set the timeout to 2x the P50 recovery time.
5. Challenge: Design a resilience system for an e-commerce checkout service that calls: (1) inventory service (check stock) (2) payment gateway (Process payment) (3) shipping service (calculate rates) and (4) notification service (send confirmation). Each dependency must have a circuit breaker with the appropriate failure threshold, recovery timeout, and fallback. When the payment circuit is open, Queue payments for retry and return a "payment pending" response. When shipping is down, return estimated rates based on default pricing. Circuit State must be monitored and alerted via a health dashboard.
Mini Project: Circuit Breaker Dashboard
# circuit_dashboard.py
# Monitoring dashboard for circuit breaker states
import time
from datetime import datetime
from collections import defaultdict
class CircuitMonitor:
"""Monitor circuit breaker states across services."""
def __init__(self):
self.circuits = {}
self.events = []
def Register_circuit(self, name: str, breaker):
"""Register a circuit breaker for monitoring."""
self.circuits[name] = breaker
def record_event(self, circuit_name: str, event_type: str, details: str = ""):
"""Record a circuit State change event."""
self.events.append({
'circuit': circuit_name,
'event': event_type,
'details': details,
'timestamp': datetime.utcnow().isoformat(),
})
def get_status(self) -> dict:
"""Get the status of all registered circuits."""
status = {}
for name, breaker in self.circuits.items():
status[name] = {
'State': breaker.State.value if hasattr(breaker.State, 'value') else breaker.State,
'failure_count': breaker.failure_count if hasattr(breaker, 'failure_count') else breaker.failureCount,
'last_failure': datetime.fromtimestamp(
breaker.last_failure_time if hasattr(breaker, 'last_failure_time')
else (breaker.lastFailureTime / 1000 if breaker.lastFailureTime else 0)
).isoformat() if (hasattr(breaker, 'last_failure_time') and breaker.last_failure_time) or
(hasattr(breaker, 'lastFailureTime') and breaker.lastFailureTime) else None,
'healthy': breaker.State in ('CLOSED', 'closed'),
}
return status
def generate_report(self):
"""Print circuit breaker status report."""
status = self.get_status()
open_circuits = [n for n, s in status.items() if not s['healthy']]
recent_events = self.events[-10:] if len(self.events) > 10 else self.events
print("=== Circuit Breaker Dashboard ===")
print(f"Total circuits: {len(status)}")
print(f"Healthy: {len(status) - len(open_circuits)}")
print(f"Open/Failed: {len(open_circuits)}")
print()
print("Circuit States:")
for name, info in status.items():
icon = "[OK]" if info['healthy'] else "[!!]"
print(f" {icon} {name}: {info['State']} "
f"(failures: {info['failure_count']})")
if open_circuits:
print(f"\nWARNING: Open circuits: {', '.join(open_circuits)}")
print()
print("Recent Events:")
for event in recent_events[-5:]:
print(f" {event['timestamp']} | {event['circuit']} | {event['event']}")
monitor = CircuitMonitor()
monitor.record_event('payment-API', 'CIRCUIT_OPEN', '5 consecutive failures')
monitor.record_event('shipping-API', 'CIRCUIT_CLOSED', 'Service recovered')
monitor.record_event('inventory-db', 'CIRCUIT_HALF_OPEN', 'Probe request sent')
monitor.generate_report()
FAQ
Related Concepts
What's Next
You now understand the Circuit Breaker Pattern. Next, learn about retry strategies for transient error handling, then explore health check endpoints for monitoring service health.
- Practice daily — Add a circuit breaker to an external API call in your application
- Build a project — Build a circuit breaker dashboard that monitors multiple services, displays State transitions, and alerts on open circuits
- Explore related topics — Check out bulkhead patterns and resilience engineering
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro