# The First Hour of an Incident

> Three roles and one shared document. Mitigate first, understand second.

Source: https://christwentyfour.com/blog/the-first-hour-of-an-incident

Small teams skip incident process because they think they are too small for it. Then four people chase one theory, nobody updates the customer, and nobody can rebuild the timeline afterward. You need three roles and one shared place to write. On a small team, one person can hold two roles.

## Coordinate from outside the system

The person debugging cannot also track who is doing what, make the escalation call, and update everyone. Give coordination to someone else. They need to know what each person is working on and catch duplicated effort.

## Update on a schedule

Post an update every fifteen minutes, even if it says "no change." The schedule stops people from asking one at a time. The quiet in between is the goal.

## Write it down as it happens

Record each action and observation in one channel. It stops repeated fixes and lets late joiners catch up. Afterward, it is the only honest timeline. Memory turns events into a neat story that is usually wrong.

## Mitigate first, understand second

Restore service, then find the cause. Roll back the recent change even if you are not sure it caused the problem. Fail over. Turn the feature off. Save logs and a snapshot before you restart anything.

The math agrees. Availability depends on time to restore as much as on how often you fail:

$$A = \frac{\mathrm{MTBF}}{\mathrm{MTBF} + \mathrm{MTTR}}$$

**Worked example:** **99.9%** every 500 h, 30 min restore · **99.8%** every 250 h

Fail every 500 hours and restore in 30 minutes: \(500 / 500.5 = 99.9\%\). Fail twice as often, every 250 hours, and you get 99.8%. Cutting restore time to 15 minutes brings it back to 99.9%, and is usually cheaper than preventing every failure.

## Declare early

If calling an incident and standing it down after ten minutes costs nothing, people will call it on time instead of forty minutes late.