E-commerce Platform Design

Monolith vs Microservices: The Real Decision

Most Magento stores run as a monolith. This is not a failure — it is often the correct choice. The decision framework:

Stay Monolith When:
├── Team < 15 engineers
├── Traffic < 50K orders/day
├── Single brand/store
├── Budget < $500K/year infrastructure
└── Need fast feature iteration

Extract Services When:
├── Team > 20 engineers (parallel work blocked)
├── Specific component needs independent scaling (search, checkout)
├── Regulatory separation required (PCI scope reduction)
├── Multiple brands with shared backend
└── Component failure must not affect catalog browsing

Real trade-off: Checkout extraction. A major retailer extracted checkout into a separate service to reduce PCI scope. Result: checkout latency increased 40ms (extra network hop), but PCI audit cost dropped $200K/year. The break-even was 18 months. Was it worth it? Only if checkout failure rate was high enough to justify the complexity.

Database Strategy: SQL vs NoSQL vs Hybrid

MySQL (Primary):
├── Order data, customer data, catalog metadata
├── ACID compliance for financial transactions
├── Complex joins for reporting
└── Limitation: Write contention at >10K orders/hour

Elasticsearch:
├── Product search, category browsing
├── Full-text search with facets
├── Aggregations for analytics
└── Limitation: Eventual consistency (stale results possible)

Redis:
├── Session data, cart state, cache
├── Sub-millisecond reads
├── Pub/Sub for real-time events
└── Limitation: Memory-bound, no complex queries

MongoDB (Optional):
├── Product reviews, CMS content, logs
├── Schema flexibility for evolving content
├── Horizontal scaling for write-heavy workloads
└── Limitation: No ACID across collections

Real failure scenario: A store used Redis for cart storage without persistence. Redis restarted during a deployment, 2,000 active carts were lost. Customer service was flooded. Fix: Enable Redis AOF persistence + periodic snapshots. Cost: 15% more memory. Impact: Zero cart loss on restart.

Synchronous vs Asynchronous Processing

Sync (Must complete before response):
├── Payment authorization
├── Inventory reservation
├── Order creation
└── Risk: If any step slow, entire request slow

Async (Can complete after response):
├── Email sending
├── Analytics tracking
├── Search index update
├── Inventory sync to warehouse
└── Risk: Eventual consistency, replay needed on failure

Production incident: A store sent order confirmation emails synchronously. During a campaign, email service latency spiked to 8 seconds. Checkout conversion dropped 35%. Fix: Move email to message queue (RabbitMQ). Queue backed up but checkout remained fast. Recovery: Replay queue after email service recovered. Business impact: $0 lost revenue vs estimated $50K/hour during the incident.

Horizontal Scaling: When and How

Scale Vertically When:
├── Single request is CPU/memory intensive
├── Database is bottleneck (queries, not connections)
├── Cost of coordination > cost of hardware
└── Example: Upgrade DB server from 64GB to 128GB RAM

Scale Horizontally When:
├── Requests are independent (stateless)
├── Database can handle read distribution
├── Load balancer cost < single server cost
└── Example: Add web nodes behind HAProxy

Real cost analysis:

Option A: Single server (256GB RAM, 32 cores)
├── Cost: $2,000/month (bare metal)
├── Capacity: 10K concurrent users
└── Single point of failure

Option B: 3 servers (64GB RAM, 8 cores each)
├── Cost: $900/month ($300 each)
├── Capacity: 12K concurrent users (4K each)
├── High availability: 1 failure = 8K capacity
└── Network overhead: ~5ms per request

Option C: Cloud auto-scaling (4-8 instances)
├── Cost: $600-1200/month (variable)
├── Capacity: 16K-32K concurrent users
├── No capacity planning needed
└── Risk: Cold start latency (30-60 seconds)

The right answer depends on traffic pattern. Steady traffic: Option B. Flash sales: Option C. Budget constrained: Option A.

Caching Strategy: Multi-Level Analysis

Cache Level     | TTL    | Hit Rate | Cost    | Risk
────────────────|────────|─────────|─────────|────────────
Browser/CDN     | 24hrs  | 85%     | Low     | Stale content
Varnish         | 1hr    | 90%     | Medium  | Cache stampede
Redis           | 30min  | 95%     | Medium  | Memory pressure
DB Query Cache  | 5min   | 60%     | Low     | Invalidation complexity

Real failure: Cache stampede. A store had 1-hour TTL on product pages. At the top of every hour, 50K requests hit the database simultaneously. Response time spiked from 100ms to 5 seconds. Fix: Add jitter to TTL (±10 minutes) + stale-while-revalidate pattern. Result: Peak DB load dropped 80%.

Search Architecture: Elasticsearch Sizing

Sizing Rules of Thumb:
├── Index size: 1KB per product (with attributes)
├── RAM: 1GB per 1M products
├── Shards: 1 per 50GB index
├── Replicas: 1 minimum (read scaling)
└── Query latency target: < 100ms

Real sizing for 100K products:
├── Index size: ~100MB
├── RAM needed: ~2GB
├── 1 shard, 1 replica
├── 2 nodes minimum (HA)
└── Cost: $200/month (cloud)

Real sizing for 1M products:
├── Index size: ~1GB
├── RAM needed: ~8GB
├── 3 shards, 1 replica
├── 3 nodes minimum
└── Cost: $600/month (cloud)

Production incident: A store with 500K products used 1 shard with 8GB RAM. During reindex, search queries timed out for 3 minutes. Customer-facing impact: search returned empty results. Fix: Increase to 3 shards, add dedicated master node, implement reindex-throttling. Cost increase: $150/month. Downtime prevented: ~3 minutes per reindex.

Single Points of Failure

Critical SPOFs in Magento:
├── Database (if single instance)
│   ├── Failure: Complete store down
│   ├── Mitigation: Master-slave replication
│   └── Cost: +50% database cost
├── Redis (if single instance)
│   ├── Failure: Sessions lost, cache miss storm
│   ├── Mitigation: Redis Sentinel or Cluster
│   └── Cost: +100% Redis cost
├── Elasticsearch (if single instance)
│   ├── Failure: Search returns empty/broken
│   ├── Mitigation: Multi-node cluster
│   └── Cost: +200% search cost
└── Payment Gateway (external)
    ├── Failure: Checkout blocked
    ├── Mitigation: Secondary gateway fallback
    └── Cost: Second merchant account fees

Cascading Failures

Real scenario: Redis fails → all cache misses → database gets 50x normal load → database slow → PHP-FPM processes pile up → web server exhausted → entire store down.

Prevention chain:

1. Circuit breaker on Redis
   └── If Redis down > 30s, serve from DB (degraded mode)
2. Database connection pooling
   └── Limit concurrent connections to 100
   └── Queue excess requests (max 50)
3. Graceful degradation
   └── Disable non-essential features (reviews, recommendations)
   └── Serve cached pages from Varnish (even if stale)
4. Rate limiting
   └── Max 100 requests/second per IP
   └── Max 1000 requests/second total

Deployment Failures

Real incident: Store deployed code that had a PHP syntax error in a helper class. The helper was only used in admin order grid. But the error occurred during setup:di:compile, which blocks deployment. Store continued running old code for 6 hours until someone noticed admin was broken.

Prevention:

1. Pre-deployment checks:
   ├── PHP lint check on all files
   ├── Static analysis (phpstan level 5+)
   ├── Unit tests on changed files
   └── Integration tests on critical paths
2. Deployment strategy:
   ├── Blue-green: Deploy to inactive environment
   ├── Smoke test: Hit 5 critical endpoints
   ├── Traffic switch: Load balancer weight
   └── Rollback: < 30 seconds
3. Post-deployment:
   ├── Monitor error rate for 30 minutes
   ├── Check order completion rate
   ├── Verify search is returning results
   └── Alert if any metric deviates > 20%

Data Corruption Scenarios

Real scenario: A cron job updated product prices but had a bug that set all prices to $0.00. The job ran for 15 minutes before detection. 2,000 orders were placed at $0.

Prevention:

1. Price validation rules:
   ├── Reject price < $0.01
   ├── Reject price change > 50% without approval
   └── Log all bulk price updates
2. Order safeguards:
   ├── Flag orders with total < $1 for review
   ├── Limit order quantity per customer
   └── CAPTCHA on checkout during price anomalies
3. Monitoring:
   ├── Alert if average order value drops > 30%
   ├── Alert if price update affects > 1000 products
   └── Real-time dashboard of order values

Three Pillars of Observability

Logs:
├── Application logs (PHP error_log, monolog)
├── Web server logs (access, error)
├── Database logs (slow query, error)
├── Search logs (indexing, query)
└── Tool: ELK Stack or Grafana Loki

Metrics:
├── Response time (p50, p95, p99)
├── Error rate (4xx, 5xx)
├── Throughput (requests/second)
├── Saturation (CPU, memory, disk, connections)
├── Business metrics (orders/hour, revenue/hour)
└── Tool: Prometheus + Grafana

Traces:
├── Request flow through services
├── Database query timing
├── Cache hit/miss ratio
├── External API latency
└── Tool: Jaeger or OpenTelemetry

Key Metrics for Magento

Business Metrics:
├── Conversion rate (target: > 2%)
├── Cart abandonment rate (target: < 70%)
├── Average order value
├── Revenue per hour
└── Search success rate

Technical Metrics:
├── TTFB (target: < 200ms)
├── FCP (target: < 1.5s)
├── LCP (target: < 2.5s)
├── Error rate (target: < 0.1%)
├── Cache hit ratio (target: > 90%)
└── Database query time (target: < 50ms avg)

Infrastructure Metrics:
├── CPU usage (alert: > 80% sustained)
├── Memory usage (alert: > 85%)
├── Disk usage (alert: > 90%)
├── Database connections (alert: > 80% pool)
└── Redis memory (alert: > 80% maxmemory)

Alerting Strategy

Severity 1 (Page immediately):
├── Store completely down
├── Payment processing failing
├── Order completion rate drops > 50%
└── Database replication lag > 30 seconds

Severity 2 (Page within 15 minutes):
├── Response time p99 > 3 seconds
├── Error rate > 1%
├── Search returning empty results
└── Cache hit ratio < 70%

Severity 3 (Notify, investigate during business hours):
├── Response time p95 > 1 second
├── Disk usage > 85%
├── Slow query count increasing
└── SSL certificate expiring in 30 days

Alert Fatigue Prevention:
├── Require 3 consecutive breaches before page
├── Auto-resolve alerts after 30 minutes stable
├── Weekly alert review meeting
└── Monthly alert tuning session

Real example: A store had 47 active alerts. Engineers ignored all of them. During an actual outage, the critical alert was buried. Fix: Reduce to 8 alerts, each with clear action runbooks. Response time to incidents improved from 45 minutes to 8 minutes.