Alerting

Alert Rule Structure

Rule: High Error Rate
  Condition: error_rate > 5% for 5 minutes
  Severity: Critical
  Action: Page on-call engineer
  Runbook: https://wiki/runbook/error-rate

Prometheus Alert Rules

# alerts.yml
groups:
  - name: magento
    rules:
      - alert: HighErrorRate
        expr: rate(magento_http_requests_total{status=~"5.."}[5m])
              / rate(magento_http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"
          runbook: "https://wiki/runbook/high-error-rate"

      - alert: HighLatency
        expr: histogram_quantile(0.99,
                rate(magento_http_request_duration_seconds_bucket[5m]))
              > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High P99 latency"
          description: "P99 latency is {{ $value }}s"

Alert Severity Levels

Severity    | Response Time | Notification    | Example
────────────|───────────────|─────────────────|─────────────────
Critical    | 5 minutes     | Page + Call     | Site down
High        | 15 minutes    | Page + Slack    | Error rate spike
Medium      | 1 hour        | Slack + Email   | High latency
Low         | Next business | Email           | Disk 80% full

Escalation Flow

Level 1: On-call engineer (0-15 min)
    │ Not acknowledged
    â–¼
Level 2: Secondary on-call (15-30 min)
    │ Not resolved
    â–¼
Level 3: Team lead (30-60 min)
    │ Not resolved
    â–¼
Level 4: Engineering manager (1+ hours)
    │ Not resolved
    â–¼
Level 5: VP Engineering (emergency)

PagerDuty Escalation

{
  "escalation_policy": {
    "name": "Magento Production",
    "escalation_rules": [
      {
        "escalation_delay": 0,
        "targets": [{ "type": "user", "id": "oncall-primary" }]
      },
      {
        "escalation_delay": 900,
        "targets": [{ "type": "user", "id": "oncall-secondary" }]
      },
      {
        "escalation_delay": 1800,
        "targets": [{ "type": "user", "id": "team-lead" }]
      }
    ]
  }
}

Acknowledgment Rules

- Auto-acknowledge after 15 minutes (escalation)
- Acknowledge before escalation to stop flow
- Resolve requires verification of fix
- Snooze max 4 hours with justification

Fatigue Symptoms

1. Alerts ignored or auto-acknowledged
2. Response time increasing
3. False positives accepted as normal
4. Critical alerts missed in noise

Fatigue Prevention

Strategy              | Implementation
──────────────────────|──────────────────────────────
Reduce noise          | Combine related alerts
Actionable alerts     | Only alert on actionable items
Threshold tuning      | Adjust based on baseline
Alert deduplication   | Don't re-alert same issue
Time-based routing    | Route by time of day
Context-rich alerts    | Include runbook link

Alert Quality Checklist

â–¡ Is the alert actionable?
â–¡ Is there a runbook?
â–¡ Is the threshold appropriate?
â–¡ Is the severity correct?
â–¡ Is the routing correct?
â–¡ Does it require human intervention?
â–¡ Can it be auto-remediated?

Alert Audit

# Review alert history
# Find noisy alerts (firing > 10 times/day)
SELECT alert_name, COUNT(*) as fires
FROM alert_history
WHERE fired_at > NOW() - INTERVAL 1 DAY
GROUP BY alert_name
HAVING fires > 10
ORDER BY fires DESC;

On-Call Rotation

Schedule: 1 week shifts
Handoff: Monday 10 AM
Backup: Secondary on-call

Responsibilities:
- Monitor alerts
- Respond to incidents
- Escalate if needed
- Document incidents

On-Call Runbook

# On-Call Responsibilities

## Start of Shift
1. Review open incidents
2. Check monitoring dashboards
3. Verify alert routing

## During Shift
1. Respond to alerts within SLA
2. Acknowledge within 5 minutes
3. Investigate and resolve
4. Escalate if needed
5. Document actions taken

## End of Shift
1. Handoff to next on-call
2. Summarize open issues
3. Update incident tickets

Incident Response

1. Acknowledge alert (5 min)
2. Assess impact (5 min)
3. Mitigate (15 min)
   - Rollback if deployment
   - Scale if capacity
   - Disable feature if bug
4. Investigate root cause
5. Resolve and verify
6. Post-incident review