Skip to content
advanced Phase 120 · Capstone

Capstone - Production Operations

Capstone production operations - deployment, monitoring, scaling, incident response, and operational excellence

3h
0 problems
Topic Progress 0%

CI/CD Pipeline Design

Pipeline Architecture

Code Push → GitHub Actions → Build → Test → Deploy → Verify
    │            │             │       │       │        │
    │            │             │       │       │        └─ Health checks
    │            │             │       │       └─ Staging → Production
    │            │             │       └─ Unit, Integration, Functional
    │            │             └─ Composer, NPM, Static deploy
    │            └─ Lint, Security scan
    └─ Feature branch, PR

GitHub Actions Workflow

# .github/workflows/deploy.yml
name: Magento CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  MAGENTO_VERSION: '2.4.7'
  PHP_VERSION: '8.2'

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: PHP CodeSniffer
        run: |
          composer install
          vendor/bin/phpcs --standard=PSR12 app/code/Vendor/
      - name: PHPStan
        run: vendor/bin/phpstan analyse app/code/Vendor/ --level=8

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Composer Audit
        run: composer audit
      - name: Security Checker
        run: |
          composer global require sensiolabs/security-checker
          security-checker security:check composer.lock

  unit-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: ${{ env.PHP_VERSION }}
          extensions: mbstring, intl, bcmath, gd
      - name: Install Dependencies
        run: composer install --no-interaction
      - name: Run Tests
        run: vendor/bin/phpunit --configuration phpunit.xml.dist

  integration-test:
    runs-on: ubuntu-latest
    needs: [unit-test]
    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: root
          MYSQL_DATABASE: magento_test
        ports:
          - 3306:3306
      redis:
        image: redis:7
        ports:
          - 6379:6306
      opensearch:
        image: opensearchproject/opensearch:2.11.0
        ports:
          - 9200:9200
    steps:
      - uses: actions/checkout@v4
      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: ${{ env.PHP_VERSION }}
      - name: Install Magento
        run: |
          composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition .
          bin/magento setup:install \
            --db-host=127.0.0.1 \
            --db-name=magento_test \
            --db-user=root \
            --db-password=root \
            --admin-firstname=admin \
            --admin-lastname=admin \
            --admin-email=admin@example.com \
            --admin-user=admin \
            --admin-password=admin123 \
            --search-engine=opensearch
      - name: Run Integration Tests
        run: vendor/bin/phpunit --configuration dev/tests/integration/phpunit.xml.dist

  deploy-staging:
    runs-on: ubuntu-latest
    needs: [lint, security, unit-test, integration-test]
    if: github.ref == 'refs/heads/develop'
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to Staging
        run: |
          ssh ${{ secrets.STAGING_SSH }} "cd /var/www/magento && \
            git pull origin develop && \
            composer install --no-dev && \
            bin/magento setup:upgrade --keep-generated && \
            bin/magento setup:di:compile && \
            bin/magento setup:static-content:deploy -f && \
            bin/magento cache:clean"
      - name: Health Check
        run: |
          sleep 30
          curl -sf https://staging.magento.local/health || exit 1

  deploy-production:
    runs-on: ubuntu-latest
    needs: [lint, security, unit-test, integration-test]
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to Production
        run: |
          # Blue-green deployment
          ssh ${{ secrets.PROD_SSH }} "cd /var/www/magento-green && \
            git pull origin main && \
            composer install --no-dev && \
            bin/magento setup:upgrade --keep-generated && \
            bin/magento setup:di:compile && \
            bin/magento setup:static-content:deploy -f && \
            bin/magento cache:clean"
      - name: Switch Traffic
        run: |
          ssh ${{ secrets.PROD_SSH }} "sudo nginx -s reload"
      - name: Verify Deployment
        run: |
          sleep 30
          curl -sf https://magento.local/health || exit 1
      - name: Rollback on Failure
        if: failure()
        run: |
          ssh ${{ secrets.PROD_SSH }} "sudo nginx -s reload -c /etc/nginx/nginx-blue.conf"

Deployment Scripts

#!/bin/bash
# deploy.sh

set -e

ENVIRONMENT=$1
VERSION=$2

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$ENVIRONMENT] $1"
}

log "Starting deployment v$VERSION"

# Step 1: Enable maintenance mode
log "Enabling maintenance mode..."
bin/magento maintenance:enable

# Step 2: Backup current state
log "Creating backup..."
bin/magento setup:backup --code --db

# Step 3: Pull code
log "Pulling code..."
git fetch origin
git checkout $VERSION

# Step 4: Install dependencies
log "Installing dependencies..."
composer install --no-dev --prefer-dist --no-interaction

# Step 5: Run migrations
log "Running setup:upgrade..."
bin/magento setup:upgrade --keep-generated

# Step 6: Compile DI
log "Compiling dependency injection..."
bin/magento setup:di:compile

# Step 7: Deploy static content
log "Deploying static content..."
bin/magento setup:static-content:deploy -f

# Step 8: Clean cache
log "Cleaning cache..."
bin/magento cache:clean
bin/magento cache:flush

# Step 9: Reindex
log "Reindexing..."
bin/magento indexer:reindex

# Step 10: Disable maintenance mode
log "Disabling maintenance mode..."
bin/magento maintenance:disable

# Step 11: Verify deployment
log "Verifying deployment..."
bin/magento --version

log "Deployment complete!"

Rollback Script

#!/bin/bash
# rollback.sh

set -e

BACKUP_ID=$1

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ROLLBACK] $1"
}

log "Starting rollback to backup $BACKUP_ID"

# Enable maintenance mode
bin/magento maintenance:enable

# Restore from backup
bin/magento setup:rollback --backup-id $BACKUP_ID

# Restore database
bin/magento db:backup:restore --backup-id $BACKUP_ID

# Clean cache
bin/magento cache:clean
bin/magento cache:flush

# Disable maintenance mode
bin/magento maintenance:disable

log "Rollback complete!"

Monitoring and Alerting

Monitoring Stack Configuration

# docker-compose.monitoring.yml
version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - '9090:9090'

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    ports:
      - '3000:3000'
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - '9093:9093'

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - '9100:9100'

  mysql-exporter:
    image: prom/mysqld-exporter:latest
    ports:
      - '9104:9104'

  redis-exporter:
    image: oliver006/redis_exporter:latest
    ports:
      - '9121:9121'

volumes:
  prometheus_data:
  grafana_data:

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'rules/*.yml'

scrape_configs:
  - job_name: 'magento'
    static_configs:
      - targets: ['web-1:9100', 'web-2:9100']

  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql-exporter:9104']

  - job_name: 'redis'
    static_configs:
      - targets: ['redis-exporter:9121']

  - job_name: 'php-fpm'
    static_configs:
      - targets: ['web-1:9253', 'web-2:9253']

Alerting Rules

# rules/magento-alerts.yml
groups:
  - name: magento-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'High error rate detected'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High response latency'
          description: 'P99 latency is {{ $value }}s'

      - alert: LowCacheHitRate
        expr: redis_keyspace_hits / (redis_keyspace_hits + redis_keyspace_misses) < 0.7
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Low cache hit rate'
          description: 'Cache hit rate is {{ $value | humanizePercentage }}'

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal - node_memory_MemAvailable) / node_memory_MemTotal > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'High memory usage'
          description: 'Memory usage is {{ $value | humanizePercentage }}'

      - alert: MySQLReplicationLag
        expr: mysql_slave_status_seconds_behind_master > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'MySQL replication lag'
          description: 'Replication lag is {{ $value }}s'

      - alert: QueueBacklog
        expr: rabbitmq_queue_messages > 5000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Queue backlog building up'
          description: '{{ $labels.queue }} has {{ $value }} messages'

Grafana Dashboard JSON

{
  "dashboard": {
    "title": "Magento Production Overview",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [{
          "expr": "rate(http_requests_total[1m])",
          "legendFormat": "{{method}} {{status}}"
        }],
        "thresholds": [{"value": 5000, "color": "red"}]
      },
      {
        "title": "Response Time",
        "type": "graph",
        "targets": [{
          "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
          "legendFormat": "P99 Latency"
        }],
        "thresholds": [{"value": 2, "color": "red"}]
      },
      {
        "title": "Error Rate",
        "type": "singlestat",
        "targets": [{
          "expr": "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m]) * 100"
        }],
        "thresholds": [
          {"value": 1, "color": "yellow"},
          {"value": 5, "color": "red"}
        ]
      },
      {
        "title": "Cache Hit Rate",
        "type": "singlestat",
        "targets": [{
          "expr": "redis_keyspace_hits / (redis_keyspace_hits + redis_keyspace_misses) * 100"
        }],
        "thresholds": [
          {"value": 70, "color": "yellow"},
          {"value": 90, "color": "green"}
        ]
      }
    ]
  }
}

Incident Response

Incident Response Runbook

# Incident Response Runbook

## Severity Levels

| Level | Description              | Response Time | Escalation     |
|-------|--------------------------|---------------|----------------|
| SEV1  | Site down, data loss     | 15 minutes    | Immediate      |
| SEV2  | Major feature broken     | 30 minutes    | 1 hour         |
| SEV3  | Minor feature degraded   | 4 hours       | 24 hours       |
| SEV4  | Cosmetic issue           | 24 hours      | Next sprint    |

## Incident Response Process

1. **Detection** (0-5 min)
   - Alert received via PagerDuty
   - Acknowledge alert
   - Join incident channel

2. **Triage** (5-15 min)
   - Assess severity
   - Identify affected systems
   - Notify stakeholders

3. **Investigation** (15-60 min)
   - Check monitoring dashboards
   - Review logs
   - Identify root cause

4. **Mitigation** (1-2 hours)
   - Implement temporary fix
   - Restore service
   - Monitor for stability

5. **Resolution** (2-24 hours)
   - Implement permanent fix
   - Verify fix works
   - Update documentation

6. **Post-Mortem** (24-48 hours)
   - Conduct blameless post-mortem
   - Document lessons learned
   - Create action items

Common Incident Runbooks

# SEV1: Magento Site Down

## Symptoms
- 5xx errors on all pages
- Health check failing
- Users cannot access site

## Immediate Actions
1. Check load balancer health
2. Check web server status
3. Check database connectivity
4. Check Redis connectivity

## Diagnostic Commands
```bash
# Check web server
sudo systemctl status nginx
sudo systemctl status php8.2-fpm

# Check database
mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"

# Check Redis
redis-cli ping

# Check disk space
df -h

# Check memory
free -m

# Check logs
tail -100 /var/log/nginx/error.log
tail -100 /var/log/php-fpm/error.log
bin/magento cache:status

Resolution Steps

  1. If web server down: sudo systemctl restart nginx php8.2-fpm
  2. If database down: sudo systemctl restart mysql
  3. If Redis down: sudo systemctl restart redis
  4. If disk full: Clear old logs, cache
  5. If memory full: Restart services, check for leaks

```markdown
# SEV2: Slow Performance

## Symptoms
- Response time > 5 seconds
- High CPU/Memory usage
- Users complaining about slowness

## Diagnostic Commands
```bash
# Check top processes
top -bn1 | head -20

# Check PHP-FPM status
curl http://localhost/fpm-status

# Check MySQL queries
mysql -u root -p -e "SHOW PROCESSLIST;"

# Check cache hit rate
redis-cli INFO stats | grep keyspace

# Check Varnish
curl -I http://localhost:8080

Resolution Steps

  1. Identify resource bottleneck
  2. Scale web nodes if CPU high
  3. Add MySQL read replica if DB slow
  4. Clear cache if hit rate low
  5. Kill long-running queries

## Post-Mortem Template

```markdown
# Post-Mortem: [Incident Title]

## Summary
- **Date:** [Date]
- **Duration:** [Duration]
- **Severity:** [SEV1/2/3]
- **Impact:** [User impact]

## Timeline
- [Time] Alert triggered
- [Time] Incident declared
- [Time] Root cause identified
- [Time] Mitigation applied
- [Time] Service restored
- [Time] Post-mortem started

## Root Cause
[Detailed root cause analysis]

## What Went Well
- [Thing 1]
- [Thing 2]

## What Went Wrong
- [Thing 1]
- [Thing 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Action 1] | [Person] | [Date] | Open |
| [Action 2] | [Person] | [Date] | Open |

## Lessons Learned
- [Lesson 1]
- [Lesson 2]

Operational Excellence

SLA/SLO/SLI Definitions

## Service Level Objectives (SLOs)

| Metric              | SLO Target    | SLI Measurement              |
|---------------------|---------------|------------------------------|
| Availability        | 99.9%         | Uptime monitoring            |
| Latency (P99)       | < 2 seconds   | APM metrics                  |
| Error Rate          | < 0.1%        | HTTP 5xx / total requests    |
| Throughput          | > 1000 RPS    | Load balancer metrics        |
| Data Durability     | 99.999%       | Backup verification          |

## Error Budget
- Monthly availability target: 99.9%
- Allowed downtime: 43.8 minutes/month
- Current error budget: [Calculated]

Operational Checklist

## Daily Operations
- [ ] Review monitoring dashboards
- [ ] Check error rates and alerts
- [ ] Verify backup completion
- [ ] Review security logs
- [ ] Check queue depths

## Weekly Operations
- [ ] Performance trend analysis
- [ ] Capacity planning review
- [ ] Security patch assessment
- [ ] Dependency updates
- [ ] Documentation updates

## Monthly Operations
- [ ] Disaster recovery drill
- [ ] Performance load testing
- [ ] Security audit
- [ ] Cost optimization review
- [ ] Team retrospective

## Quarterly Operations
- [ ] Architecture review
- [ ] Technology upgrade planning
- [ ] Compliance audit
- [ ] Business continuity test
- [ ] Vendor review

Runbook Automation

#!/usr/bin/env python3
# auto_scale.py

import boto3
import json
from datetime import datetime, timedelta

class AutoScaler:
    def __init__(self):
        self.cloudwatch = boto3.client('cloudwatch')
        self.asg = boto3.client('autoscaling')
        self.asg_name = 'magento-web'

    def check_and_scale(self):
        metrics = self.get_metrics()

        if metrics['cpu_avg'] > 60:
            self.scale_up(2)
        elif metrics['cpu_avg'] < 30 and metrics['current'] > 2:
            self.scale_down(1)

    def get_metrics(self):
        response = self.cloudwatch.get_metric_statistics(
            Namespace='AWS/EC2',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'AutoScalingGroupName', 'Value': self.asg_name}],
            StartTime=datetime.utcnow() - timedelta(minutes=5),
            EndTime=datetime.utcnow(),
            Period=300,
            Statistics=['Average']
        )

        asg = self.asg.describe_auto_scaling_groups(
            AutoScalingGroupNames=[self.asg_name]
        )

        return {
            'cpu_avg': response['Datapoints'][0]['Average'] if response['Datapoints'] else 0,
            'current': asg['AutoScalingGroups'][0]['DesiredCapacity']
        }

    def scale_up(self, count):
        current = self.get_metrics()['current']
        new_count = min(current + count, 16)
        self.asg.set_desired_capacity(
            AutoScalingGroupName=self.asg_name,
            DesiredCapacity=new_count
        )
        print(f'Scaled up: {current} -> {new_count}')

    def scale_down(self, count):
        current = self.get_metrics()['current']
        new_count = max(current - count, 2)
        self.asg.set_desired_capacity(
            AutoScalingGroupName=self.asg_name,
            DesiredCapacity=new_count
        )
        print(f'Scaled down: {current} -> {new_count}')

if __name__ == '__main__':
    scaler = AutoScaler()
    scaler.check_and_scale()

Cost Optimization

## Cost Optimization Strategies

### Infrastructure
- Right-size instances based on actual usage
- Use Reserved Instances for predictable workloads
- Use Spot Instances for non-critical work
- Auto-scale to match traffic patterns

### Database
- Use read replicas for read-heavy workloads
- Archive old data to reduce storage
- Optimize queries to reduce CPU usage

### Cache
- Optimize cache hit rates
- Reduce cache invalidation frequency
- Use CDN for static assets

### Storage
- lifecycle policies for old media
- Compress images automatically
- Clean up unused resources

## Monthly Cost Review
| Component     | Current Cost | Optimization | Savings |
|---------------|-------------|--------------|---------|
| EC2           | $X,XXX      | Reserved     | $XXX    |
| RDS           | $X,XXX      | Right-size   | $XXX    |
| ElastiCache   | $XXX        | Cluster      | $XX     |
| S3            | $XXX        | Lifecycle    | $XX     |
| DataTransfer  | $XXX        | CDN          | $XX     |

Quiz

1. What is the purpose of CI/CD?

Question 1 options

2. What is a post-mortem?

Question 2 options

3. What is SLO?

Question 3 options

Flashcards

Question

What is CI/CD?

Click to reveal answer

Answer

Continuous Integration / Continuous Deployment - automated build/test/deploy

Question

What is a runbook?

Click to reveal answer

Answer

Step-by-step guide for handling specific incidents

Question

What is SLO?

Click to reveal answer

Answer

Service Level Objective - target metrics for service performance

Question

What is error budget?

Click to reveal answer

Answer

Allowed downtime based on SLO (e.g., 99.9% = 43.8 min/month)

Question

What is blue-green deployment?

Click to reveal answer

Answer

Two identical environments, switch traffic between them

Revision Notes

Key Takeaways

  • 1. CI/CD automates build, test, and deployment processes
  • 2. Monitoring uses Prometheus/Grafana for metrics and alerting
  • 3. Incident response follows detection → triage → investigation → resolution
  • 4. Post-mortems are blameless and focus on improvements
  • 5. Operational excellence includes SLAs, checklists, and automation

Interview Tips

  • • Describe your CI/CD pipeline and deployment strategy
  • • Explain monitoring and alerting setup
  • • Discuss incident response process and runbooks
  • • Talk about operational excellence practices

Cheat Sheet

CI/CD Pipeline:
  Lint → Security → Test → Build → Deploy → Verify
  Staging → Production (with approval)
  Rollback on failure

Monitoring:
  Prometheus → Metrics collection
  Grafana → Visualization dashboards
  Alertmanager → Alert routing
  ELK → Centralized logging

Incident Response:
  SEV1: 15min response, immediate escalation
  SEV2: 30min response, 1hr escalation
  SEV3: 4hr response, 24hr escalation
  SEV4: 24hr response, next sprint

Post-Mortem:
  Blameless analysis
  Timeline, root cause, action items
  Lessons learned

Operational Excellence:
  SLA: 99.9% availability
  SLO: <2s latency, <0.1% errors
  Error budget: 43.8 min/month
  Daily/weekly/monthly checklists