Most runbooks are written by the person who least needs them, immediately after they have solved the problem, while every piece of context is still loaded in their head. That is exactly the wrong moment, and it produces documents that are technically accurate and practically useless.
The failure mode has a name: the curse of knowledge. Once you know a thing, you cannot reconstruct what it was like not to know it. So you skip steps, because to you they stopped being steps. "Connect to the cluster and check the queue depth" is four separate pieces of knowledge pretending to be one instruction.
The test
A runbook is real if somebody else can execute it, on the worst night of the quarter, without asking you anything. That is the whole bar. Everything else is a proxy for it.
The only reliable way to know is to have somebody else run it. Not read it, run it. Pick a low-stakes window, hand it to whoever has the least context, and watch without helping. Every question they ask is a defect in the document. Write it down and fix it after.
This is uncomfortable, which is why it rarely happens. It is also the difference between a runbook and a folder of confident-sounding notes.
Write it for the reader you will actually have
Assume the reader is tired, mildly panicked, and unfamiliar with this service. That reader is often you, six months later, and you will be surprised how little you remember.
Some things that follow from that:
- Exact commands, copy-pasteable, with the placeholders obvious.
- What the output should look like when it worked, so the reader can tell.
- What to do when it did not work, including who to wake and when.
- A stop condition. "If this has not resolved within twenty minutes, escalate" is more useful than an open-ended list of things to try.
Runbooks are a byproduct, not a project
Nobody has ever successfully scheduled a quarter of runbook writing. The work does not compete well against features, and a document written from memory in a planning sprint is worse than no document, because it carries false authority.
The version that works is to write the runbook while doing the thing. You are already in the system, already have the commands in your shell history, already know which of them was the one that mattered. Ten minutes at the end of an incident, while it is still raw, produces something far better than an hour of recollection later.
Rehearse the ones that matter
For the handful of procedures where the stakes are real, restoring a database, failing over, rolling back a bad release, reading is not enough. Those get practiced, on a schedule, in an environment where being wrong is cheap. An unrehearsed recovery procedure is a hypothesis about what would happen, and the middle of an outage is a bad time to test one.