Postmortems

What is a Postmortem?

A postmortem is a written record of an incident:

1. What happened?
2. When did it happen?
3. What was the impact?
4. What was the root cause?
5. What was done to resolve it?
6. What can be done to prevent it?

Benefits:
├── Learn from incidents
├── Prevent recurrence
├── Improve processes
├── Share knowledge
└── Build culture

Blameless Culture

Blameless Postmortem Principles:

1. Focus on systems, not individuals
2. Everyone makes mistakes
3. Systems should be designed to prevent errors
4. Learning is the goal, not blame
5. Psychological safety is essential

What NOT to do:
├── Don't blame individuals
├── Don't punish mistakes
├── Don't focus on who, but what/why
└── Don't skip postmortems

When to Write Postmortems

Write postmortems when:
├── Customer-facing impact
├── Data loss or corruption
├── Security incident
├── SLA breach
├── Significant process failure
└── Learning opportunity

Don't write for:
├── Minor issues
├── Expected maintenance
├── Individual mistakes
└── External factors only

Standard Format

# Postmortem: [Incident Title]

## Metadata
- **Date:** YYYY-MM-DD
- **Author:** Name
- **Status:** Draft/Final
- **Incident ID:** INC-XXXX
- **Duration:** X hours Y minutes
- **Severity:** SEV1/SEV2/SEV3

## Executive Summary
[1-2 sentence summary of what happened and impact]

## Impact
- **Customer Impact:** [How customers were affected]
- **Business Impact:** [Revenue, metrics affected]
- **Duration:** [Start time - End time]
- **Affected Users:** [Number/percentage]

## Timeline
[Chronological list of events]

## Root Cause
[Detailed explanation of root cause]

## Resolution
[What was done to resolve the incident]

## Detection
[How was the incident detected?]

## Action Items
| Priority | Action Item | Owner | Due Date | Status |
|----------|-------------|-------|----------|--------|
| P1 | Action 1 | Name | Date | Open |
| P2 | Action 2 | Name | Date | Open |

## Lessons Learned
[What went well, what went wrong]

## Supporting Information
[Links to logs, dashboards, related incidents]

Example Postmortem

# Postmortem: Checkout Failure During Flash Sale

## Metadata
- **Date:** 2025-01-15
- **Author:** John Doe
- **Status:** Final
- **Incident ID:** INC-1234
- **Duration:** 2 hours
- **Severity:** SEV1

## Executive Summary
Checkout failed for 30% of users during flash sale due to payment gateway timeout.

## Impact
- **Customer Impact:** 30% checkout failures
- **Business Impact:** $50K lost revenue
- **Duration:** 14:00 - 16:00
- **Affected Users:** 5,000

## Timeline
- 14:00: Flash sale started
- 14:15: Traffic spike
- 14:20: Payment gateway timeouts
- 14:25: Monitoring alerts
- 14:30: Team notified
- 14:45: Circuit breaker enabled
- 15:00: Traffic reduced
- 15:30: System stabilized
- 16:00: Incident resolved

## Root Cause
Payment gateway not configured for high traffic, no rate limiting.

## Resolution
Enabled circuit breaker, rate limiting, and fallback to secondary gateway.

## Detection
Monitoring detected error rate spike.

## Action Items
| Priority | Action Item | Owner | Due Date | Status |
|----------|-------------|-------|----------|--------|
| P1 | Add rate limiting | John | 2025-01-20 | Open |
| P1 | Configure circuit breaker | Jane | 2025-01-20 | Open |
| P2 | Add load testing | Mike | 2025-01-25 | Open |
| P2 | Update runbook | Sarah | 2025-01-25 | Open |

## Lessons Learned
- Good: Monitoring detected quickly
- Good: Team responded fast
- Bad: No rate limiting in place
- Bad: Load testing not done

## Supporting Information
- Dashboard: [link]
- Logs: [link]
- Related incidents: INC-1230

Create Action Items

// Action item management
class ActionItemManager
{
    public function createFromPostmortem($postmortem)
    {
        $items = [];
        
        // Extract action items from postmortem
        foreach ($postmortem->getActionItems() as $item) {
            $actionItem = $this->actionItemFactory->create();
            $actionItem->setTitle($item['title']);
            $actionItem->setDescription($item['description']);
            $actionItem->setAssignee($item['assignee']);
            $actionItem->setDueDate($item['due_date']);
            $actionItem->setPriority($item['priority']);
            $actionItem->setIncidentId($postmortem->getIncidentId());
            $actionItem->setStatus('open');
            
            $this->actionItemRepository->save($actionItem);
            $items[] = $actionItem;
        }
        
        return $items;
    }
}

Track Action Items

// Track action item progress
class ActionItemTracker
{
    public function getOverdueItems()
    {
        return $this->actionItemRepository->getList([
            'status' => 'open',
            'due_date' => ['lt' => date('Y-m-d')]
        ]);
    }
    
    public function getProgress($incidentId)
    {
        $items = $this->actionItemRepository->getByIncident($incidentId);
        
        $total = count($items);
        $completed = count(array_filter($items, function($item) {
            return $item->getStatus() === 'completed';
        }));
        
        return [
            'total' => $total,
            'completed' => $completed,
            'percentage' => ($completed / $total) * 100
        ];
    }
}

Review Action Items

// Weekly action item review
public function weeklyReview()
{
    $overdue = $this->tracker->getOverdueItems();
    $upcoming = $this->tracker->getUpcomingItems(7); // Next 7 days
    
    // Send review email
    $this->emailService->sendActionItemReview([
        'overdue' => $overdue,
        'upcoming' => $upcoming
    ]);
    
    // Escalate overdue items
    foreach ($overdue as $item) {
        $this->escalate($item);
    }
}

Building Culture

Steps to Build Postmortem Culture:

1. Leadership Support
   ├── Leaders participate in postmortems
   ├── Allocate time for postmortems
   └── Celebrate learning

2. Blameless Environment
   ├── Focus on systems, not individuals
   ├── Encourage honesty
   └── Protect psychological safety

3. Regular Practice
   ├── Schedule postmortems promptly
   ├── Include all relevant parties
   └── Follow up on action items

4. Knowledge Sharing
   ├── Share postmortems widely
   ├── Extract lessons learned
   └── Update documentation

5. Continuous Improvement
   ├── Review postmortem process
   ├── Refine templates
   └── Celebrate improvements

Facilitation

// Postmortem facilitation
class PostmortemFacilitator
{
    public function facilitate($incident)
    {
        $meeting = [
            'duration' => 60, // minutes
            'attendees' => $this->getAttendees($incident),
            'agenda' => [
                '0-5 min': 'Read-only review of timeline',
                '5-20 min': 'Discussion of what happened',
                '20-40 min': 'Root cause analysis',
                '40-55 min': 'Action items',
                '55-60 min': 'Next steps'
            ]
        ];
        
        return $meeting;
    }
    
    private function getAttendees($incident)
    {
        return [
            'incident commander',
            'technical lead',
            'on-call engineer',
            'affected team members',
            'optional: leadership'
        ];
    }
}

Common Mistakes

Mistakes to Avoid:

1. Blaming Individuals
   ❌ "John broke the deploy"
   ✅ "The deployment process allowed the error"

2. Skipping Postmortems
   ❌ "It was minor, no need"
   ✅ "Every incident is a learning opportunity"

3. No Follow-up
   ❌ "Action items created but not tracked"
   ✅ "Track and review action items regularly"

4. Too Long/Too Short
   ❌ 3-hour meetings
   ❌ 5-minute discussions
   ✅ 60-90 minute focused sessions

5. Not Sharing
   ❌ Postmortem in private folder
   ✅ Share with team and organization

Track Metrics

// Postmortem metrics
$metrics = [
    'total_postmortems' => $this->getPostmortemCount(),
    'action_items_completed' => $this->getCompletedItems(),
    'action_items_overdue' => $this->getOverdueItems(),
    'avg_time_to_postmortem' => $this->getAvgTimeToPostmortem(),
    'avg_time_to_resolve' => $this->getAvgTimeToResolve(),
    'incident_frequency' => $this->getIncidentFrequency()
];

// Goals
$goals = [
    'action_item_completion_rate' => 90, // percent
    'time_to_postmortem' => 48, // hours
    'incident_reduction' => 20 // percent per quarter
];

Improvement Tracking

// Track improvements
public function trackImprovements()
{
    $quarterly = [
        'Q1' => ['incidents' => 10, 'postmortems' => 8, 'improvements' => 5],
        'Q2' => ['incidents' => 8, 'postmortems' => 7, 'improvements' => 6],
        'Q3' => ['incidents' => 5, 'postmortems' => 5, 'improvements' => 4],
        'Q4' => ['incidents' => 3, 'postmortems' => 3, 'improvements' => 3]
    ];
    
    // Calculate trends
    $incidentTrend = $this->calculateTrend($quarterly, 'incidents');
    $improvementTrend = $this->calculateTrend($quarterly, 'improvements');
    
    return [
        'incident_trend' => $incidentTrend,
        'improvement_trend' => $improvementTrend,
        'is_improving' => $incidentTrend < 0 // Negative means decreasing
    ];
}