Disaster Recovery

DR Levels: Cost vs Recovery

Level 1: Backup & Restore
├── RPO: 24 hours, RTO: 24-48 hours
├── Cost: $100-500/month (storage + backup)
├── Best for: Small stores, non-critical
└── Risk: Up to 24 hours of data loss

Level 2: Pilot Light
├── RPO: minutes, RTO: 1-4 hours
├── Cost: $500-2000/month (minimal running infra)
├── Best for: Medium stores, moderate revenue
└── Risk: Manual activation required

Level 3: Warm Standby
├── RPO: seconds, RTO: minutes
├── Cost: $2000-10000/month (scaled-down duplicate)
├── Best for: High-revenue stores, strict SLA
└── Risk: Data consistency during failover

Level 4: Active-Active
├── RPO: 0, RTO: instant
├── Cost: $10000-50000/month (full duplicate)
├── Best for: Enterprise, 24/7 global operations
└── Risk: Complexity, data conflicts

Real decision framework:

Annual Revenue: $10M
Hourly Revenue: $10M / 8760 = $1,142/hour
Downtime Cost: $1,142/hour × hours = total loss

Level 1 (24hr RTO):
├── Cost: $300/month = $3,600/year
├── Risk: 24 hours × $1,142 = $27,408 potential loss
└── ROI: 7.6x if disaster occurs once/year

Level 2 (4hr RTO):
├── Cost: $1,500/month = $18,000/year
├── Risk: 4 hours × $1,142 = $4,568 potential loss
└── ROI: 0.25x — only worth it if disasters frequent

Level 3 (30min RTO):
├── Cost: $5,000/month = $60,000/year
├── Risk: 0.5 hours × $1,142 = $571 potential loss
└── ROI: 0.01x — expensive insurance

Decision: Level 2 is optimal for $10M store.
Unless: Regulatory requirement or SLA penalty > $5K/hour

Real Failure Scenarios

Scenario 1: Database Corruption

Event: Disk failure corrupted MySQL data files
Impact: Store completely down
Detection: Health check failed at 2:00 AM
Recovery: Restored from 6-hour-old backup
Data Loss: 6 hours of orders (120 orders, ~$18,000)
RTO Actual: 4.5 hours
RPO Actual: 6 hours

What went wrong:
├── Backups were hourly but retention was poor
├── No binary log shipping to DR site
├── Corruption was not detected for 30 minutes
└── Recovery procedure was untested

What was fixed:
├── Enable MySQL binary log shipping (RPO → 0)
├── Add replication lag monitoring
├── Monthly DR testing
└── Automated corruption detection

Scenario 2: Redis Cluster Failure

Event: Power supply failure in Redis rack
Impact: All sessions lost, cache miss storm
Detection: User reports of being logged out
Recovery: Redis restarted from AOF, sessions lost
Data Loss: 5,000 active sessions, all cart data
RTO Actual: 15 minutes (Redis restart)
RPO Actual: 5 minutes (AOF snapshot interval)

What went wrong:
├── No Redis persistence to disk
├── No session replication
├── Cart data only in Redis (no DB backup)
└── No graceful degradation when Redis down

What was fixed:
├── Enable Redis AOF with 1-second fsync
├── Session backup to database every 5 minutes
├── Cart data persisted to DB on every change
└── Circuit breaker: serve degraded mode if Redis > 30s down

Scenario 3: Payment Gateway Outage

Event: Stripe had regional outage (2 hours)
Impact: All credit card payments failing
Detection: Checkout error rate spiked to 100%
Recovery: Switched to PayPal fallback
Data Loss: None (orders queued)
RTO Actual: 8 minutes (manual switch)
RPO Actual: 0

What went wrong:
├── No automatic gateway failover
├── No health check on payment gateway
├── Customers saw generic error message
└── No queue for failed payment attempts

What was fixed:
├── Implement gateway health checks (ping every 30s)
├── Auto-switch to PayPal if Stripe fails 3 consecutive requests
├── Queue failed payment attempts for retry
└── Show customer-friendly message with alternative payment

Backup Types and Timing

Full Backup:
├── Complete database dump
├── Frequency: Daily (2:00 AM low-traffic)
├── Size: 10GB for typical store
├── Time: 30-60 minutes
├── Cost: $50/month storage
└── Risk: Restores from last full only

Incremental Backup:
├── Changes since last backup
├── Frequency: Hourly
├── Size: 100MB-1GB per backup
├── Time: 5-15 minutes
├── Cost: $20/month storage
└── Risk: Must restore full + all incrementals

Binary Log Shipping:
├── Real-time MySQL binlog replication
├── Frequency: Continuous
├── Size: 1-5GB/day
├── Time: Near-zero (streaming)
├── Cost: $100/month storage + bandwidth
└── Risk: Complex setup, point-in-time recovery

Filesystem Backup:
├── Media files, configuration, code
├── Frequency: Daily + on deploy
├── Size: 50-500GB
├── Time: 1-4 hours
├── Cost: $100-500/month storage
└── Risk: Large restores, versioning complexity

Backup Verification

Monthly verification checklist:
├── [ ] Restore database backup to test server
├── [ ] Verify data integrity (record counts, checksums)
├── [ ] Run application against restored data
├── [ ] Measure restore time vs RTO target
├── [ ] Document any issues found
└── [ ] Update runbook with lessons learned

Real incident: Backup verification revealed that 3 months of backups were corrupted. The backup script had a silent failure. No one noticed because verification was never done. Fix: Automated daily verification + alerting on backup failure.

Geo-Redundancy Decisions

Same Region (us-east-1):
├── Cost: +20% (cross-AZ replication)
├── RPO: seconds
├── RTO: minutes
├── Risk: Region-wide outage still affects both
└── Best for: Most stores

Different Region (us-east-1 → us-west-2):
├── Cost: +50-100% (cross-region bandwidth)
├── RPO: minutes (async replication)
├── RTO: 30-60 minutes (DNS propagation)
├── Risk: Data consistency, higher latency
└── Best for: Regulatory compliance, global stores

Different Continent (US → EU):
├── Cost: +100-200% (infrastructure + compliance)
├── RPO: hours (large replication lag)
├── RTO: hours (complex activation)
├── Risk: GDPR, latency, time zones
└── Best for: Multi-continent operations

Real decision: A US-based store with 5% EU customers chose same-region DR. Reason: EU customers were not revenue-critical, and cross-region cost ($3K/month extra) exceeded EU revenue impact ($1K/month). Regulatory compliance was not required for their EU sales model.

Testing Types and Frequency

Tabletop Exercise (Monthly):
├── Duration: 1-2 hours
├── Participants: Engineering leads, ops, business
├── Activity: Walk through DR scenario on paper
├── Cost: $0 (just time)
├── Value: Process gaps, role clarity
└── Output: Updated runbook, action items

Component Test (Quarterly):
├── Duration: 4-8 hours
├── Participants: Engineering team
├── Activity: Failover individual components
├── Cost: $500 (临时 infrastructure)
├── Value: Validate individual failover mechanisms
└── Output: Component-level recovery metrics

Partial DR Test (Semi-Annual):
├── Duration: 1-2 days
├── Participants: Full team
├── Activity: Failover subset of systems
├── Cost: $2000 (temporary DR environment)
├── Value: End-to-end recovery validation
└── Output: Recovery time, data loss measurement

Full DR Test (Annual):
├── Duration: 2-3 days
├── Participants: Full team + stakeholders
├── Activity: Complete failover to DR site
├── Cost: $5000-10000 (full DR environment + lost revenue)
├── Value: Complete DR validation
└── Output: Full recovery report, RTO/RPO actuals

DR Test Script with Real Metrics

#!/bin/bash
# DR Test Procedure with Timing

START_TIME=$(date +%s)
echo "=== DR Test Start: $(date) ==="

# Phase 1: Backup current state
echo "Phase 1: Backup production"
time mysqldump -u root magento_db > /tmp/pre-dr-test-backup.sql
PHASE1_TIME=$(($(date +%s) - START_TIME))
echo "Phase 1 Duration: ${PHASE1_TIME}s"

# Phase 2: Simulate primary failure
echo "Phase 2: Simulate failure"
# Stop writes to primary (simulate crash)
mysql -u root -e "SET GLOBAL read_only = ON;"
sleep 10  # Allow replication to catch up
PHASE2_TIME=$(($(date +%s) - START_TIME - PHASE1_TIME))
echo "Phase 2 Duration: ${PHASE2_TIME}s"

# Phase 3: Activate DR
echo "Phase 3: Activate DR"
# Promote DR replica
ssh dr-server "mysql -u root -e 'STOP SLAVE; RESET SLAVE ALL;'"
# Update DNS
dns_update --record store.example.com --value dr.example.com
# Wait for DNS propagation
sleep 30
PHASE3_TIME=$(($(date +%s) - START_TIME - PHASE1_TIME - PHASE2_TIME))
echo "Phase 3 Duration: ${PHASE3_TIME}s"

# Phase 4: Verify application
echo "Phase 4: Verify"
curl -f https://store.example.com/health_check.php
RESPONSE=$(curl -s -o /dev/null -w '%{http_code}' https://store.example.com/)
if [ "$RESPONSE" = "200" ]; then
    echo "Application: OK"
else
    echo "Application: FAILED (HTTP $RESPONSE)"
fi
PHASE4_TIME=$(($(date +%s) - START_TIME - PHASE1_TIME - PHASE2_TIME - PHASE3_TIME))
echo "Phase 4 Duration: ${PHASE4_TIME}s"

# Phase 5: Measure metrics
echo "Phase 5: Metrics"
TOTAL_TIME=$(($(date +%s) - START_TIME))
echo "Total Recovery Time: ${TOTAL_TIME}s"
echo "RTO Target: 3600s (1 hour)"
echo "RTO Actual: ${TOTAL_TIME}s"
if [ $TOTAL_TIME -le 3600 ]; then
    echo "RTO: PASS"
else
    echo "RTO: FAIL"
fi

# Measure data loss
echo "=== DR Test Complete ==="

Post-DR Test Review

Questions to answer:
├── Did we meet RTO target? If not, what was slow?
├── Did we meet RPO target? What data was lost?
├── Were there steps that failed or were unclear?
├── Did communication work as expected?
├── Were there unexpected dependencies?
└── What would we do differently next time?

Real example: DR test revealed that DNS propagation took 45 minutes instead of expected 5. Fix: Implement health-check-based DNS failover (Route53 health checks) instead of manual DNS update. Reduced DNS failover from 45 minutes to 60 seconds.

Incident Severity and Response

Severity 1 (Store Down):
├── Response: Page on-call immediately
├── Communication: Status page updated in 5 minutes
├── Decision: Activate DR if not recovered in 30 minutes
├── Escalation: CTO within 15 minutes
└── Post-incident: Blameless postmortem within 48 hours

Severity 2 (Major Feature Broken):
├── Response: Page on-call within 15 minutes
├── Communication: Status page updated in 15 minutes
├── Decision: Activate DR if payment or checkout affected
├── Escalation: Engineering lead within 30 minutes
└── Post-incident: Review within 1 week

Severity 3 (Degraded Performance):
├── Response: Investigate during business hours
├── Communication: Internal notification only
├── Decision: No DR activation
├── Escalation: Engineering lead within 2 hours
└── Post-incident: Document in weekly review

DR Activation Decision Tree

Is store completely down?
├── YES → Is database accessible?
│   ├── YES → Restore from backup (Level 1)
│   └── NO → Activate DR site (Level 2+)
└── NO → Is checkout broken?
    ├── YES → Activate DR if payment gateway affected
    └── NO → Is performance degraded > 50%?
        ├── YES → Scale horizontally, monitor
        └── NO → Investigate, no DR needed

Runbook Template with Decision Points

# Magento DR Runbook

## Pre-Conditions
- [ ] DR environment is healthy (last check: ___)
- [ ] Backups are current (last backup: ___)
- [ ] Team availability confirmed

## Activation Decision
- Store down > 30 minutes? → YES/NO
- Payment processing failing? → YES/NO
- Revenue impact > $1000/hour? → YES/NO
- Decision: ACTIVATE DR / DO NOT ACTIVATE

## Activation Steps
1. [ ] Declare DR event (who: ___)
2. [ ] Notify stakeholders (template below)
3. [ ] Promote DR database (command: ___)
4. [ ] Update DNS (method: manual/automated)
5. [ ] Verify application (URL: ___)
6. [ ] Run smoke tests (script: ___)
7. [ ] Monitor for 30 minutes

## Rollback Decision
- Is production healthy? → YES/NO
- Is DR performing as expected? → YES/NO
- Decision: STAY ON DR / ROLLBACK TO PRODUCTION

## Communication Templates

### Initial Notification
Subject: [Severity X] Magento Store - Service Disruption
Status: Investigating
Impact: [Description]
Next update: [Time]

### Resolution
Subject: [Resolved] Magento Store - Service Restored
Duration: [X hours Y minutes]
Root cause: [Brief description]
Action items: [List]

Common Runbook Mistakes

Mistake 1: Runbook not updated
├── Scenario: Runbook references old DNS records
├── Discovery: During actual DR activation
├── Impact: 30 minutes wasted updating DNS manually
└── Prevention: Monthly runbook review + automated validation

Mistake 2: No rollback plan
├── Scenario: DR activated but didn't work
├── Discovery: During DR test
├── Impact: Stuck in DR with no way back
└── Prevention: Always include rollback steps

Mistake 3: Unclear decision authority
├── Scenario: Two people disagree on DR activation
├── Discovery: During actual incident
├── Impact: 15 minutes of debate during outage
└── Prevention: Pre-assign decision authority by severity level