---
title: Postmortems
slug: postmortems
summary: A blameless postmortem template and a fully worked fictional example, so an agent can draft one from an incident timeline in minutes.
order: 8
updated: 2026-11-11
status: ready
---

Use this page after any SEV1 or SEV2, any near miss that could have been one, and any incident that happened twice. A postmortem exists to change the system so the same thing cannot hurt users again. It is not a report card on people.

## Rules for a useful postmortem

- [ ] Blameless: describe what people saw and why their action made sense at the time. Use roles, not names.
- [ ] Draft within 2 business days while memory is fresh. Review within 5.
- [ ] Every action item has an owner, a ticket and a due date.
- [ ] Prefer fixes that remove a class of failure (automation, guardrails) over "be more careful".
- [ ] No secrets, customer data or raw logs with identifiers in the document.

## How an agent drafts one

1. Collect the timeline from the incident channel, alert history, deploy history and CloudTrail (read-only).
2. Convert every time to UTC and sort.
3. Fill the template below. Mark anything inferred as `(inferred)`.
4. Propose 3 to 6 action items, ranked by how much risk each removes.
5. Hand the draft to the incident commander. Do not publish it yourself.

## Template

```markdown
## Summary
<2-3 sentences: what broke, who was affected, for how long, how it was fixed.>

## Impact
- Duration: <start> to <end> UTC (<N> min)
- Users affected: <count or percent, segment>
- SLO impact: <budget consumed>
- Data impact: <none / describe>

## Timeline (UTC)
| Time | Event |
| --- | --- |
| HH:MM | First bad data point |
| HH:MM | Alert fired / report received |
| HH:MM | Incident declared, IC assigned |
| HH:MM | Mitigation applied |
| HH:MM | Resolved |

## Root cause and contributing factors
<What happened, in causal order. Ask "why" until you reach something you can change.>

## Detection
<How we found out. Could we have found out sooner? Did monitoring work?>

## What went well
## What went poorly
## Where we got lucky

## Action items
| Action | Type | Owner | Ticket | Due |
| --- | --- | --- | --- | --- |
| | prevent / detect / mitigate / process | | PROJ-123 | |
```

## Worked example (fictional)

### Summary

For 47 minutes, about 30 percent of API requests to the orders service returned 503. A new release passed smoke tests, but the production configuration pointed the service at a connection pool size that the database could not serve under peak load. Rolling back the release restored service.

### Impact

- Duration: 14:02 to 14:49 UTC (47 min)
- Users affected: roughly 30 percent of order submissions failed; clients retried, so 4 percent were lost
- SLO impact: 65 percent of the monthly availability budget consumed
- How that was worked out: \(\text{budget used} = \dfrac{\text{duration} \times \text{error fraction}}{\text{budget}}\). With a 99.95 percent SLO the 30-day budget is 21.6 minutes, and \(47 \times 0.30 = 14.1\) minutes is 65 percent of it.
- Data impact: none; failed requests were never committed

### Timeline (UTC)

| Time | Event |
| --- | --- |
| 13:55 | Release promoted to production |
| 14:02 | 5xx rate rises from 0.1 percent to 28 percent |
| 14:09 | Fast-burn SLO alert pages on-call |
| 14:12 | Incident declared, SEV2, IC assigned |
| 14:21 | Deploy history shows the 13:55 release; rollback proposed |
| 14:38 | Rollback approved and run |
| 14:49 | Error rate at baseline, incident resolved |

### Root cause and contributing factors

- The release raised the per-task database pool from 10 to 50. With 20 tasks at peak, that exceeded the database connection limit.
- Smoke tests ran on one candidate task at low load, so they could not catch it.
- The config change lived in the same MR as a code change, and review focused on the code.
- Rollback took 17 minutes after it was proposed because nobody on the call had prod deploy rights.

### Action items

| Action | Type | Owner | Ticket | Due |
| --- | --- | --- | --- | --- |
| Alert when database connections exceed 80 percent of the limit | detect | Service team | PROJ-123 | +7 d |
| Pipeline check: total pool size times max tasks under the database limit | prevent | Platform team | PROJ-124 | +14 d |
| Give on-call the prod rollback permission | mitigate | Platform lead | PROJ-125 | +3 d |
| Separate config-only changes into their own MR | process | Service team | PROJ-126 | +14 d |

## Pitfalls

- "Human error" as a root cause. Ask what made the error easy.
- Action items with no owner. They never happen.
- Postmortems nobody reads. Share a three-line summary in the team channel and review open action items monthly.