Skip to content
advanced Phase 86 · Distributed Advanced

Circuit Breakers

Circuit breaker pattern with state machine, threshold configuration, and fallback mechanisms

45m
0 problems
Topic Progress 0%

Circuit Breaker States

State Machine

┌──────────┐     Threshold     ┌──────────┐
│  CLOSED  │ ────────────────▶  │   OPEN   │
│ (Normal) │   failures hit    │ (Blocked)│
└──────────┘                   └──────────┘
     ▲                              │
     │        Recovery timeout      │
     └──────────────────────────────┘
              HALF-OPEN
          (Testing recovery)

State Transitions

CLOSED → OPEN: Failure count >= threshold
OPEN → HALF-OPEN: Recovery timeout elapsed
HALF-OPEN → CLOSED: Probe succeeds
HALF-OPEN → OPEN: Probe fails

Implementation

class CircuitBreaker {
    private string $state = 'closed';
    private int $failureCount = 0;
    private int $successCount = 0;
    private int $lastFailureTime = 0;
    
    public function __construct(
        private int $failureThreshold = 5,
        private int $recoveryTimeout = 60,
        private int $halfOpenMaxCalls = 3
    ) {}
    
    public function call(callable $fn) {
        $this->checkState();
        
        if ($this->state === 'open') {
            throw new CircuitOpenException('Circuit is open');
        }
        
        try {
            $result = $fn();
            $this->onSuccess();
            return $result;
        } catch (Exception $e) {
            $this->onFailure();
            throw $e;
        }
    }
    
    private function onSuccess() {
        if ($this->state === 'half-open') {
            $this->successCount++;
            if ($this->successCount >= $this->halfOpenMaxCalls) {
                $this->state = 'closed';
                $this->failureCount = 0;
            }
        } else {
            $this->failureCount = 0;
        }
    }
    
    private function onFailure() {
        $this->failureCount++;
        $this->lastFailureTime = time();
        
        if ($this->failureCount >= $this->failureThreshold) {
            $this->state = 'open';
        }
    }
    
    private function checkState() {
        if ($this->state === 'open') {
            if (time() - $this->lastFailureTime > $this->recoveryTimeout) {
                $this->state = 'half-open';
                $this->successCount = 0;
            }
        }
    }
}

Threshold Configuration

Threshold Types

Type              | Description                | Default
──────────────────|────────────────────────────|─────────
Failure count     | Number of failures         | 5
Failure rate      | Percentage of failures     | 50%
Slow call rate    | Percentage of slow calls   | 80%
Slow call threshold | Duration considered slow | 2s

Service-Specific Config

$configs = [
    'payment' => [
        'failure_threshold' => 3,
        'recovery_timeout' => 30,
        'half_open_max_calls' => 2
    ],
    'search' => [
        'failure_threshold' => 5,
        'recovery_timeout' => 15,
        'half_open_max_calls' => 3
    ],
    'email' => [
        'failure_threshold' => 3,
        'recovery_timeout' => 60,
        'half_open_max_calls' => 2
    ]
];

Adaptive Thresholds

// Adjust thresholds based on traffic
function getFailureThreshold($service) {
    $traffic = $this->metrics->getTrafficRate($service);
    
    if ($traffic > 1000) {
        return 10; // Higher threshold for high traffic
    } elseif ($traffic > 100) {
        return 5;
    } else {
        return 3; // Lower threshold for low traffic
    }
}

Fallback Mechanisms

Fallback Strategies

Strategy           | Description              | Use Case
───────────────────|──────────────────────────|──────────────
Return cached data | Return last known good   | Read operations
Return default     | Return default value     | Non-critical
Queue for retry    | Async processing later   | Write operations
Graceful degradation | Disable feature       | Non-essential

Fallback Chain

function search($query) {
    try {
        return $this->circuitBreaker['search']->call(
            fn() => $this->opensearch->search($query)
        );
    } catch (CircuitOpenException $e) {
        // Fallback 1: Try cache
        try {
            return $this->cache->get('search_' . md5($query));
        } catch (Exception $e) {
            // Fallback 2: Default results
            return $this->getDefaultResults();
        }
    }
}

Payment Fallback

function processPayment($order) {
    try {
        return $this->circuitBreaker['payment']->call(
            fn() => $this->stripe->charge($order)
        );
    } catch (CircuitOpenException $e) {
        // Fallback: Queue for retry
        $this->queue->publish('payment.retry', $order->toArray());
        $this->orderService->markPendingPayment($order);
        return ['status' => 'pending', 'message' => 'Payment processing'];
    }
}

Monitoring Circuit Breakers

Metrics to Track

Metric              | Description              | Alert
────────────────────|──────────────────────────|──────────
State               | closed/open/half-open    | On open
Failure count       | Current failures         | >80% threshold
Success rate        | Calls succeeding         | <90%
Fallback rate       | Fallbacks triggered      | >10%
Recovery time       | Time to close            | >60s

Dashboard

// Expose circuit breaker metrics
$metrics = [];
foreach ($this->breakers as $name => $breaker) {
    $metrics[$name] = [
        'state' => $breaker->getState(),
        'failure_count' => $breaker->getFailureCount(),
        'success_rate' => $breaker->getSuccessRate(),
        'fallback_rate' => $breaker->getFallbackRate()
    ];
}
return $metrics;

Alerting

# Alert when circuit opens
alert: CircuitBreakerOpen
expr: circuit_breaker_state{state="open"} == 1
for: 1m
labels:
  severity: critical
annotations:
  summary: "Circuit breaker {{ $labels.service }} is OPEN"

Quiz

1. When does a circuit breaker open?

Question 1 options

2. What happens in half-open state?

Question 2 options

3. What is the purpose of fallbacks?

Question 3 options

Flashcards

Question

Circuit breaker states?

Click to reveal answer

Answer

Closed (normal), Open (blocked), Half-Open (testing)

Question

State transitions?

Click to reveal answer

Answer

Closed→Open: threshold; Open→Half-Open: timeout; Half-Open→Closed: probe success

Question

Fallback strategies?

Click to reveal answer

Answer

Cached data, default values, queue for retry, feature disable

Question

Monitoring metrics?

Click to reveal answer

Answer

State, failure count, success rate, fallback rate

Revision Notes

Key Takeaways

  • 1. Circuit breaker prevents cascading failures with threshold-based opening
  • 2. Three states: Closed (normal), Open (blocked), Half-Open (testing)
  • 3. Configure failure threshold and recovery timeout per service
  • 4. Fallbacks provide alternative when circuit is open
  • 5. Monitor circuit state and alert when open

Interview Tips

  • • Explain circuit breaker state machine and transitions
  • • Discuss threshold configuration strategies
  • • Describe fallback patterns and when to use each

Cheat Sheet

Circuit Breaker:
  Closed: Normal operation, count failures
  Open: Block calls, wait for timeout
  Half-Open: Test with limited calls

Thresholds:
  Failure count: 3-10
  Recovery timeout: 15-60s
  Half-open max calls: 2-3

Fallbacks:
  Cached data: Last known good
  Default: Fallback value
  Queue: Async retry
  Disable: Feature off

Monitoring:
  State: closed/open/half-open
  Failure count, success rate, fallback rate