Skip to content
advanced Phase 86 · Distributed Advanced

Retries and Backoff

Retry strategies with exponential backoff, jitter, retry limits, and retry policies

45m
0 problems
Topic Progress 0%

Exponential Backoff

Backoff Concept

Retry 1: 100ms
Retry 2: 200ms
Retry 3: 400ms
Retry 4: 800ms
Retry 5: 1600ms

Each retry doubles the wait time

Implementation

function retryWithBackoff(callable $fn, $maxRetries = 5) {
    $attempt = 0;
    
    while (true) {
        try {
            return $fn();
        } catch (TransientException $e) {
            $attempt++;
            if ($attempt >= $maxRetries) {
                throw $e;
            }
            $delay = pow(2, $attempt) * 100; // 200, 400, 800...
            usleep($delay * 1000);
        }
    }
}

// Usage
$result = retryWithBackoff(function () use ($payment) {
    return $this->gateway->charge($payment);
}, 5);

Backoff Formula

delay = base_delay * 2^attempt

base_delay = 100ms
attempt 1: 200ms
attempt 2: 400ms
attempt 3: 800ms
attempt 4: 1600ms
attempt 5: 3200ms

Jitter

Why Jitter?

Without jitter (thundering herd):
All clients retry at: 100, 200, 400, 800ms
→ All hit server simultaneously

With jitter (spread load):
Client 1: 120ms, 230ms, 510ms...
Client 2: 80ms, 310ms, 720ms...
Client 3: 150ms, 180ms, 630ms...
→ Load spread over time

Jitter Strategies

Type              | Formula                        | Distribution
──────────────────|────────────────────────────────|─────────────
Full jitter       | random(0, delay)               | Uniform
Equal jitter      | delay/2 + random(0, delay/2)   | Half uniform
Decorrelated      | random(base, prev_delay * 3)   | Random range

Implementation

function retryWithJitter(callable $fn, $maxRetries = 5) {
    $attempt = 0;
    $previousDelay = 0;
    
    while (true) {
        try {
            return $fn();
        } catch (TransientException $e) {
            $attempt++;
            if ($attempt >= $maxRetries) {
                throw $e;
            }
            
            // Full jitter
            $delay = pow(2, $attempt) * 100;
            $jitteredDelay = random_int(0, $delay);
            
            // Decorrelated jitter
            // $jitteredDelay = random_int(100, $previousDelay * 3);
            
            usleep($jitteredDelay * 1000);
            $previousDelay = $jitteredDelay;
        }
    }
}

Retry Limits and Policies

Retry Policy Configuration

$retryPolicy = [
    'max_retries' => 3,
    'base_delay_ms' => 100,
    'max_delay_ms' => 5000,
    'backoff_multiplier' => 2,
    'jitter' => true,
    'retryable_exceptions' => [
        ConnectionException::class,
        TimeoutException::class,
        ServiceUnavailableException::class
    ],
    'non_retryable_exceptions' => [
        AuthenticationException::class,
        ValidationException::class
    ]
];

Retry Decision Matrix

Exception Type         | Retry? | Strategy
───────────────────────|────────|─────────────────────
Connection timeout     | Yes    | Exponential backoff
Service unavailable    | Yes    | Exponential backoff
Rate limited (429)     | Yes    | Respect Retry-After
Authentication failed  | No     | Fail immediately
Validation error       | No     | Fail immediately
Insufficient funds     | No     | Fail immediately

Retry Limits

Service              | Max Retries | Total Timeout
─────────────────────|─────────────|───────────────
Payment gateway      | 3           | 10s
Email service        | 5           | 30s
Search service       | 2           | 5s
Inventory check      | 2           | 3s

Retry Strategies

Strategy Comparison

Strategy           | When to Use              | Example
───────────────────|──────────────────────────|─────────────
Immediate          | Rare failures            | Cache miss
Linear backoff     | Moderate load            | API calls
Exponential        | High load, recovery      | Service calls
Exponential+jitter | Thundering herd          | All distributed

Composite Retry Strategy

function resilientCall($service, $method, $args) {
    return retryWithBackoff(
        fn() => $this->callWithTimeout($service, $method, $args),
        $this->getRetryPolicy($service)
    );
}

function callWithTimeout($service, $method, $args) {
    return $this->circuitBreaker->call(
        fn() => $service->$method(...$args)
    );
}

Monitoring Retries

// Track retry metrics
$retryCounter->inc(['service' => $service, 'attempt' => $attempt]);

$retryHistogram->observe($attempt, ['service' => $service]);

// Alert on high retry rates
if ($retryRate > 0.1) { // >10% retry rate
    $this->alertService->warning('High retry rate', [
        'service' => $service,
        'rate' => $retryRate
    ]);
}

Quiz

1. Why add jitter to retries?

Question 1 options

2. Should you retry validation errors?

Question 2 options

3. What is exponential backoff?

Question 3 options

Flashcards

Question

Exponential backoff?

Click to reveal answer

Answer

Delay doubles each retry: 100ms, 200ms, 400ms, 800ms...

Question

Jitter purpose?

Click to reveal answer

Answer

Randomize retry timing to prevent thundering herd

Question

Retry limits?

Click to reveal answer

Answer

Max retries (3-5) and max total timeout per service

Question

Non-retryable errors?

Click to reveal answer

Answer

Validation, authentication - fail immediately

Revision Notes

Key Takeaways

  • 1. Exponential backoff doubles delay between retries
  • 2. Jitter randomizes timing to prevent thundering herd
  • 3. Set retry limits and total timeout per service
  • 4. Don't retry validation or authentication errors
  • 5. Monitor retry rates and alert on anomalies

Interview Tips

  • • Explain exponential backoff and jitter strategies
  • • Discuss retry decision matrix for different error types
  • • Describe monitoring and alerting for retry patterns

Cheat Sheet

Retries:
  Exponential backoff: delay * 2^attempt
  Jitter: random(0, delay) prevents thundering herd
  Limits: max 3-5 retries, total timeout

Retry Decision:
  Retry: Timeout, connection, 5xx, 429
  No retry: 4xx validation, auth, business errors

Monitoring:
  Track retry rate per service
  Alert if >10% retry rate
  Monitor p99 retry count