# What I optimize for, and what I give up to get it

> Good strategy tells you what to give up. These are the trade-offs I use when an incident or a deadline forces a call.

Source: https://christwentyfour.com/

## What a strategy is

A strategy picks the trade-offs. It isn't a roadmap or a list of projects. A good one lets any engineer make the right call without escalating.

It has three parts (Rumelt, *Good Strategy/Bad Strategy*):

1. **Diagnosis**: an honest read of where you are, including constraints and risks.
2. **Guiding principle**: the trade-off you pick, e.g. "reliability over feature count when we miss the SLA."
3. **Coherent actions**: a few moves that put the principle to work and reinforce each other.

### A strategy is a set of values

In chess, a bad strategy values pawns more than queens. A good one values queens more than pawns. Neither tells you which move to make. With the right values, you can work out the move yourself, in any position.

Engineering works the same way. Say "a deploy is worth more than a dollar of cost, as long as you stay above the SLO" and the next decision follows from it.

### Good strategy vs bad strategy

| Good | Bad |
| --- | --- |
| Names the real problem in one sentence | Restates the goal ("be more reliable") |
| Picks a trade-off and says no to things | Tries to do everything |
| A few actions that fit together | A pile of unrelated projects and OKRs |
| Plain words | Jargon |
| Gets revisited when the situation changes | Gets written once and filed |

### Strategy vs tactics

|  | Strategy | Tactics |
| --- | --- | --- |
| Question | Where do we spend effort? | How do we do this step? |
| Horizon | Quarters | Minutes to days |
| Aided by | **Models and values**: equations, charts, what each thing is worth | **Discipline, training and docs**: runbooks, checklists, practice |
| Fails when | The model is wrong | A step gets skipped |
| Example | Ship as many changes as the SLO allows | Roll back a bad deploy in under 5 minutes |

Rule:if you're deciding, reach for models and values. If you're doing, reach for discipline and a runbook.

### Why strategy matters more now

A good strategy with nobody who can execute it is worth nothing.

AI made tactics cheap. An agent can write the code, run the runbook, and draft the fix. When doing is cheap, choosing what to do is the expensive part.

## Spend your error budget on changes

Your SLO is a budget of bad minutes. Every change spends a few of them. Recover faster and each change costs less, so you can ship more.

### At 99.9%, you get 43 bad minutes a month

30-min fix

14 changes

5-min fix

86 changes

1 in 10 changes breaks something. A fix takes 30 minutes. Each change costs 3 minutes on average, so you can ship about 14 changes a month.

Same failure rate, but you roll back in 5 minutes. Each change costs 30 seconds, so you can ship about 86 a month.

If the rollback lands before the user tries again, the retry works and they never see the error.

\(\text{changes} = \text{budget} \div (\text{failure rate} \times \text{fix time})\)

43.2 ÷ (0.1 × 30) ≈ 14 · 43.2 ÷ (0.1 × 5) ≈ 86

Rule:Make changes cheap to get wrong. Halve the fix time and you double what you can ship.

## Impact: make a bad change hurt fewer people for less time

Downtime people feel is people times minutes. You can cut both without shipping any less.

10,000 users. Each small change has a 2% chance of breaking something.

### Why smaller changes break less

Every piece you bundle into a release is another chance to break something. At 2% a piece, a one-change release breaks 2 times in 100. Bundle 10 and it breaks 18 times in 100. And when it does break, you have 10 suspects instead of 1.

1 change per release

2 of 100 break

5 changes per release

10 of 100 break

10 changes per release

18 of 100 break

Each dot is one release. Gold ones broke something.

chance a release breaks = 1 − (0.98 × 0.98 × … once per change) = 1 − 0.98changes

#### One big weekly release

10 changes bundled, 18% chance it's bad

everyone · 30 min · 55,000 user-min

#### Ship them one at a time

2% each, easy to spot and roll back

everyone · 5 min · 10,000 user-min

#### Send each to 5% first

only the canary sees it

500 users · 5 min · 500 user-min

#### Retry lands on the old version

one failed click, then it works

500 users · 6 sec · 10 user-min

Each dot is 100 users. The bar is how long they notice, out of 30 minutes.

Same 10 changes, same code. The downtime people feel drops from about 55,000 user-minutes to 10.

user-minutes = changes × chance it's bad × users exposed × minutes they notice  
10 × 0.02 × 500 × 0.1 = 10

Rule:Ship small, ship to a few first, and make the retry land on the old version.

## Eliminate toil, and check that you did

Toil is repetitive manual work that leaves nothing behind. It inflates cost without adding change. Google's benchmark is under 50% of team time. The trend matters more than the threshold, because toil grows with the number of services.

**Do:**

- Automate every routine deploy through a pipeline.
- Self-heal recurring alerts. Page a human only when the fix fails.
- Make security scans pipeline stages, not periodic projects.

**Don't:**

- Fix toil by hiring. Cost then grows linearly with the problem.
- Count hours saved without checking that deploys went up.

## Catch defects in design, where they cost a conversation

A defect caught in design costs a conversation. Caught at deploy, it costs a rollback and someone's sleep. Infrastructure people belong in requirement and design reviews.

**Do:**

- Put tests in the acceptance criteria.
- Version, lint, and test config like code.
- Run static analysis and dependency scans in the PR pipeline.

### Don't lean on end-to-end tests

End-to-end tests feel like the safest test. At scale they're the least useful. Every test that has to pass is another chance to fail for a reason that has nothing to do with your code: a timeout, a stale config, a service someone else just deployed. Grow the system and the chance the whole suite passes goes to zero, even when everything is healthy. Then people re-run until it's green, and the suite stops telling you anything.

Say every test is 99% reliable on a healthy system.

10 tests

90 of 100 runs pass

50 tests

61 of 100 runs pass

100 tests

37 of 100 runs pass

300 tests

5 of 100 runs pass

Each dot is one run of the whole suite on a healthy system. Gold ones passed.

Most of those failures are config and environment, not bugs. And when one fails, it doesn't tell you which service broke. A contract test checks one service against its direct neighbors. It grows with the number of connections, not the size of the whole system, and when it fails it names the culprit.

chance the suite passes = 0.99tests  ·  0.99100 ≈ 37%

Rule:Keep a handful of end-to-end tests for the money path. Everything else is unit tests, contract tests and a canary.

## Write down what should be true

Steps don't tell you if the work is done. Checks do. Red is the work left. Green is done.

#### Steps

Done? No way to tell.

?Provision a server.

?Copy the site over.

?Point DNS at it.

#### Desired state

1 of 3 true. The red lines are the work left.

✗Exactly one web server is running.

✓example.com points at that server.

✗http redirects to https.

Rule:If you can't write the check, you don't know what done means yet.

## Attention moves from how to what

Every new tool takes over more of the steps. What's left is saying what should be true, and checking that it is.

A rough picture, not a measurement.

an>**→**All green: done