Skip to content
advanced Phase 81 · HA Fundamentals

Failover Mechanisms

Failover mechanisms including automatic failover, manual failover, health checks, and DNS failover

45m
0 problems
Topic Progress 0%

Automatic Failover

Failover Architecture

Normal:                          Failover:
┌──────────┐                    ┌──────────┐
│ Primary  │                    │ Primary  │ (DOWN)
└────┬─────┘                    └────┬─────┘
     │                               │
     ▼                          ┌────┴─────┐
┌──────────┐                    │ Replica  │ (PROMOTED)
│ Replica  │                    │ (Primary)│
└──────────┘                    └──────────┘

MySQL Replication Failover

# Promote replica to primary
mysql -u root -p -e "STOP SLAVE; RESET SLAVE ALL;"

# Update Magento config
bin/magento setup:config:set --db-host=replica-db.example.com

# Verify application connectivity
bin/magento cache:clean

Automatic Failover with Orchestrator

# Orchestrator config
{
  "DetectClusterAliasQuery": "SELECT ...",
  "DetectInstanceAliasQuery": "SELECT ...",
  "RecoverMasterClusterFilters": [".*"],
  "RecoverIntermediateMasterClusterFilters": [".*"]
}

# Orchestrator automatically:
# 1. Detects primary failure
# 2. Promotes best replica
# 3. Reconfigures other replicas
# 4. Updates application routing

Health Checks

Health Check Levels

Level 1: Process check (is service running?)
Level 2: Connectivity check (can we connect?)
Level 3: Functional check (does it work?)
Level 4: Performance check (is it fast enough?)

Magento Health Check

// pub/health_check.php
header('Content-Type: application/json');

$checks = [];

// Database check
try {
    $pdo = new PDO($dsn, $user, $pass);
    $pdo->query('SELECT 1');
    $checks['database'] = 'healthy';
} catch (Exception $e) {
    $checks['database'] = 'unhealthy';
}

// Redis check
try {
    $redis = new Redis();
    $redis->connect('redis-host', 6379);
    $redis->ping();
    $checks['redis'] = 'healthy';
} catch (Exception $e) {
    $checks['redis'] = 'unhealthy';
}

// OpenSearch check
try {
    $response = $client->request('GET', '/_cluster/health');
    $checks['opensearch'] = 'healthy';
} catch (Exception $e) {
    $checks['opensearch'] = 'unhealthy';
}

$healthy = !in_array('unhealthy', $checks);
http_response_code($healthy ? 200 : 503);

echo json_encode([
    'status' => $healthy ? 'healthy' : 'unhealthy',
    'checks' => $checks
]);

Load Balancer Health Check

# NGINX health check
location /health {
    access_log off;
    return 200 'OK';
    add_header Content-Type text/plain;
}

# HAProxy health check
backend magento_nodes
    option httpchk GET /health_check.php
    http-check expect status 200
    server web1 10.0.1.10:8080 check inter 5s fall 3 rise 2

DNS Failover

DNS Failover Architecture

store.example.com → Primary (us-east-1)
                   → Secondary (us-west-2)

If primary unhealthy → DNS routes to secondary
TTL: 60 seconds (fast failover)

Route53 Health Check

{
  "HealthCheckConfig": {
    "IPAddress": "primary.example.com",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health_check.php",
    "FailureThreshold": 3,
    "RequestInterval": 10,
    "EnableSNI": true
  }
}

DNS Failover Config

# Primary record
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "store.example.com",
        "Type": "A",
        "SetIdentifier": "primary",
        "Failover": "PRIMARY",
        "TTL": 60,
        "ResourceRecords": [{"Value": "1.2.3.4"}],
        "HealthCheckId": "abc-123"
      }
    }]
  }'

# Secondary record
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "store.example.com",
        "Type": "A",
        "SetIdentifier": "secondary",
        "Failover": "SECONDARY",
        "TTL": 60,
        "ResourceRecords": [{"Value": "5.6.7.8"}],
        "HealthCheckId": "def-456"
      }
    }]
  }'

Failover Testing

Testing Procedures

1. Simulate primary failure
2. Verify automatic failover triggers
3. Measure failover time
4. Verify data consistency
5. Test rollback procedures
6. Document findings

Failover Test Script

#!/bin/bash
# Test failover

# 1. Record current state
PRIMARY=$(mysql -e "SHOW SLAVE STATUS\G" | grep Master_Host)
echo "Current primary: $PRIMARY"

# 2. Stop primary
systemctl stop mysql

# 3. Monitor failover
for i in {1..30}; do
    NEW_PRIMARY=$(mysql -e "SHOW SLAVE STATUS\G" | grep Master_Host)
    if [ "$NEW_PRIMARY" != "$PRIMARY" ]; then
        echo "Failover completed in ${i}s"
        break
    fi
    sleep 1
done

# 4. Verify application works
curl -f http://store.example.com/health_check.php

# 5. Restart original primary
systemctl start mysql

# 6. Verify re-sync
mysql -e "SHOW SLAVE STATUS\G" | grep Slave_SQL_Running

Failover Metrics

Metric                | Target
──────────────────────|──────────────
Failover time         | <60 seconds
Data loss             | 0 (async) or minimal
Service interruption  | <5 seconds
Verification time     | <5 minutes

Quiz

1. What is the recommended DNS TTL for fast failover?

Question 1 options

2. How many health check failures before failover?

Question 2 options

3. What is the target failover time?

Question 3 options

Flashcards

Question

Health check levels?

Click to reveal answer

Answer

Process → Connectivity → Functional → Performance

Question

DNS TTL for failover?

Click to reveal answer

Answer

60 seconds for fast propagation

Question

Failover time target?

Click to reveal answer

Answer

<60 seconds for automatic failover

Question

Health check failure threshold?

Click to reveal answer

Answer

3 consecutive failures before failover

Revision Notes

Key Takeaways

  • 1. Automatic failover requires health checks and promotion logic
  • 2. Health checks should test connectivity and functionality
  • 3. DNS failover with 60-second TTL enables geographic redundancy
  • 4. Failover should complete within 60 seconds
  • 5. Regular failover testing validates HA architecture

Interview Tips

  • • Explain automatic failover mechanisms for database and web tier
  • • Discuss health check design and failure thresholds
  • • Describe DNS failover and geographic redundancy

Cheat Sheet

Failover:
  Automatic: Orchestrator for MySQL
  DNS: Route53 health checks, 60s TTL
  Load Balancer: Backend health probes

Health Checks:
  Level 1-4: Process → Connectivity → Functional → Performance
  Threshold: 3 failures
  Interval: 5-10 seconds

Testing:
  Simulate failure → Measure time → Verify data → Document
  Target: <60s failover, 0 data loss