Skip to content
advanced Phase 82 · HA Advanced

Failure Domains

Failure domains including isolating failures, blast radius reduction, and dependency failure handling

45m
0 problems
Topic Progress 0%

Isolating Failures

Failure Domain Levels

Level 1: Process (single request failure)
Level 2: Service (single service failure)
Level 3: Node (single server failure)
Level 4: Rack (rack-level failure)
Level 5: Availability Zone (AZ failure)
Level 6: Region (regional failure)

Magento Failure Isolation

┌─────────────────────────────────────────┐
│ AZ-1                                   │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐│
│  │ Web 1   │  │ Web 2   │  │ Web 3   ││
│  └────┬────┘  └────┬────┘  └────┬────┘│
│       └────────┬────────────────┘      │
│                ▼                       │
│  ┌─────────────────────────┐           │
│  │ DB Primary + Replica    │           │
│  └─────────────────────────┘           │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│ AZ-2                                   │
│  ┌─────────┐  ┌─────────┐             │
│  │ Web 4   │  │ DB Replica│            │
│  └─────────┘  └─────────┘             │
└─────────────────────────────────────────┘

AZ-1 failure → AZ-2 continues serving

Service Isolation

// Isolate critical services
try {
    $payment = $this->paymentService->charge($order);
} catch (PaymentException $e) {
    // Payment failure doesn't affect order creation
    $this->orderService->markPendingPayment($order);
    $this->queue->publish('payment.retry', $orderData);
}

Blast Radius Reduction

Blast Radius Concepts

Large Blast Radius:              Small Blast Radius:
┌────────────────────┐           ┌────────────────────┐
│ Shared Database    │           │ DB per Service     │
│ All services fail  │           │ One fails, others  │
└────────────────────┘           │ continue           │
                                 └────────────────────┘

Reduction Strategies

1. Service Isolation: Separate databases per service
2. Feature Flags: Disable features without deployment
3. Rate Limiting: Protect services from overload
4. Bulkheads: Isolate resource pools
5. Timeouts: Prevent cascade from slow services

Feature Flag Isolation

// Disable problematic feature
if ($this->featureFlag->isEnabled('reviews_enabled')) {
    $this->reviewService->process($review);
} else {
    // Graceful degradation: queue for later
    $this->queue->publish('review.deferred', $reviewData);
}

// Emergency shutdown
if ($this->featureFlag->isEnabled('emergency_disable_payments')) {
    throw new ServiceDisabledException('Payments temporarily unavailable');
}

Dependency Failure Handling

Dependency Matrix

Dependency       | Failure Impact | Fallback
─────────────────|────────────────|──────────────────
Payment gateway  | Checkout fails  | Offline payment
Email service    | Notifications   | Queue for later
Search service   | Search broken   | Cached results
CDN              | Static files    | Origin fallback
Analytics        | Tracking lost   | Buffer locally

Dependency Circuit Breakers

// Each dependency gets its own circuit breaker
$breakers = [
    'payment' => new CircuitBreaker(threshold: 5, timeout: 60),
    'email' => new CircuitBreaker(threshold: 3, timeout: 30),
    'search' => new CircuitBreaker(threshold: 10, timeout: 30),
];

// Payment service with fallback
try {
    $result = $breakers['payment']->call(function () use ($payment) {
        return $this->gateway->charge($payment);
    });
} catch (CircuitOpenException $e) {
    return $this->fallbackPayment->process($payment);
}

Graceful Dependency Failure

// Search with fallback chain
function search($query) {
    try {
        return $this->opensearch->search($query);  // Primary
    } catch (Exception $e) {
        try {
            return $this->cache->get('search_' . md5($query));  // Cache
        } catch (Exception $e) {
            return $this->getDefaultResults();  // Default
        }
    }
}

Fault-Isolated Architecture

Architecture Principles

1. Separate failure domains (AZ, region)
2. Independent data stores per service
3. Asynchronous communication where possible
4. Timeout and circuit breaker on every call
5. Feature flags for emergency isolation
6. Bulkhead pattern for resource isolation

Bulkhead Implementation

// Separate connection pools per service
$paymentPool = new ConnectionPool('payment', [
    'max_connections' => 10,
    'timeout' => 5
]);

$searchPool = new ConnectionPool('search', [
    'max_connections' => 20,
    'timeout' => 3
]);

$emailPool = new ConnectionPool('email', [
    'max_connections' => 5,
    'timeout' => 10
]);

// Search overload doesn't affect payment

Failure Domain Testing

# Chaos engineering: inject failures
docker stop payment-service
# Verify: checkout falls back gracefully

# Network partition test
iptables -A INPUT -s payment-service -j DROP
# Verify: circuit breaker opens, fallback works

# AZ failure simulation
terraform destroy -target=module.az1
# Verify: AZ-2 continues serving

Quiz

1. What is blast radius?

Question 1 options

2. What is the bulkhead pattern?

Question 2 options

3. How does feature flag isolation help?

Question 3 options

Flashcards

Question

Failure domain levels?

Click to reveal answer

Answer

Process → Service → Node → Rack → AZ → Region

Question

Blast radius reduction?

Click to reveal answer

Answer

Service isolation, feature flags, rate limiting, bulkheads

Question

Bulkhead pattern?

Click to reveal answer

Answer

Isolate resource pools so failures don't cascade

Question

Dependency failure handling?

Click to reveal answer

Answer

Circuit breakers, timeouts, fallback chains

Revision Notes

Key Takeaways

  • 1. Failure domains range from process-level to regional
  • 2. Blast radius reduction isolates failures to small scope
  • 3. Each dependency needs circuit breaker and fallback
  • 4. Bulkhead pattern prevents resource contention cascade
  • 5. Chaos testing validates fault isolation

Interview Tips

  • • Explain failure domain levels and isolation strategies
  • • Discuss blast radius reduction techniques
  • • Describe bulkhead pattern and its implementation

Cheat Sheet

Failure Domains:
  Levels: Process → Service → Node → Rack → AZ → Region
  Goal: Limit blast radius

Blast Radius Reduction:
  Service isolation
  Feature flags
  Rate limiting
  Bulkheads
  Timeouts

Bulkhead Pattern:
  Separate connection pools per service
  Isolated resource limits
  Prevents cascade failures

Testing:
  Chaos engineering
  Fault injection
  AZ failure simulation