Fault Tolerance

Degradation Strategy

Full Functionality: All services available
├── Payment gateway down → Offer bank transfer
├── Search service down → Show cached results
├── Email service down → Queue for later
├── CDN down → Serve from origin
└── Full outage → Static maintenance page

Magento Degradation Config

// Fallback for payment methods
try {
    $result = $gateway->process($payment);
} catch (ServiceUnavailableException $e) {
    // Fallback: offer alternative payment
    $this->logger->warning('Payment gateway unavailable, offering fallback');
    return $this->fallbackPayment->process($payment);
}

Degradation Levels

Level 0: Full functionality
Level 1: Non-critical features disabled (reviews, recommendations)
Level 2: Search degraded (cached results only)
Level 3: Checkout simplified (standard payment only)
Level 4: Read-only mode (browsing only)
Level 5: Maintenance mode (static page)

Fallback Strategies

// 1. Cached data fallback
function getProduct($id) {
    try {
        return $this->database->fetch($id);
    } catch (Exception $e) {
        return $this->cache->get('product_' . $id);
    }
}

// 2. Default value fallback
function getRecommendations($productId) {
    try {
        return $this->mlService->getRecommendations($productId);
    } catch (Exception $e) {
        return $this->getDefaultRecommendations();
    }
}

// 3. Queue for later
function sendEmail($data) {
    try {
        return $this->emailService->send($data);
    } catch (Exception $e) {
        $this->queue->publish('email.deferred', $data);
        return true; // Don't fail user request
    }
}

Fallback Chain

Service Call → Try Primary → Catch → Try Secondary → Catch → Use Default

Example: Search
OpenSearch → Redis Cache → Static Results → Empty Results

Circuit Breaker States

┌──────────┐     Failure      ┌──────────┐
│  CLOSED  │ ──────────────▶  │   OPEN   │
│(Normal)  │                  │(Blocked) │
└──────────┘     Success      └──────────┘
     ▲                              │
     │            Timeout           │
     └──────────────────────────────┘
                  HALF-OPEN
              (Testing recovery)

Implementation

class CircuitBreaker {
    private int $failureCount = 0;
    private int $failureThreshold = 5;
    private int $recoveryTimeout = 60;
    private string $state = 'closed';
    private int $lastFailureTime = 0;

    public function call(callable $fn) {
        if ($this->state === 'open') {
            if (time() - $this->lastFailureTime > $this->recoveryTimeout) {
                $this->state = 'half-open';
            } else {
                throw new CircuitOpenException('Circuit is open');
            }
        }

        try {
            $result = $fn();
            $this->onSuccess();
            return $result;
        } catch (Exception $e) {
            $this->onFailure();
            throw $e;
        }
    }

    private function onSuccess() {
        $this->failureCount = 0;
        $this->state = 'closed';
    }

    private function onFailure() {
        $this->failureCount++;
        $this->lastFailureTime = time();
        if ($this->failureCount >= $this->failureThreshold) {
            $this->state = 'open';
        }
    }
}

Magento Circuit Breaker Usage

$breaker = $this->circuitBreakerPool->get('payment_gateway');

try {
    $result = $breaker->call(function () use ($payment) {
        return $this->gateway->charge($payment);
    });
} catch (CircuitOpenException $e) {
    // Fallback: offline payment
    return $this->offlinePayment->process($payment);
}

Resilience Checklist

1. Timeouts on all external calls
2. Retry with exponential backoff
3. Circuit breakers for critical services
4. Graceful degradation for failures
5. Fallback mechanisms for each dependency
6. Health checks for all services
7. Monitoring and alerting

Timeout Configuration

// Set timeouts for all external services
$client = new GuzzleHttpClient([
    'timeout' => 5,           // Connection timeout
    'connect_timeout' => 3,   // Connection timeout
    'read_timeout' => 30,     // Read timeout
]);

// Redis timeout
$redis->connect('host', 6379, 2.5); // 2.5s timeout

// Database timeout
$connection = new PDO($dsn, $user, $pass, [
    PDO::ATTR_TIMEOUT => 5
]);

Bulkhead Pattern

// Isolate critical services
$paymentPool = new ConnectionPool('payment', [
    'max_connections' => 10,
    'timeout' => 5
]);

$searchPool = new ConnectionPool('search', [
    'max_connections' => 20,
    'timeout' => 3
]);

// Search failure doesn't affect payment processing