Your First SLO

Starting from one service and one number, when nobody has agreed that reliability should be measured yet.

SLOs Christopher Roos 3 min read

Most advice about service level objectives assumes you are starting from an organization that already agrees reliability should be measured. If you are not there yet, the advice is hard to act on. You do not need a company-wide programme to get value from this. You need one service and one number.

Start with the argument you keep having

The reason to define an SLO is not measurement for its own sake. It is to end a specific recurring argument: is this good enough to ship, or should we stop and fix things? Without a number, that argument gets settled by whoever is most senior or most tired, and the answer changes week to week.

So pick the service where that argument actually happens. Not the most critical one, the most contested one. That is where a number will do the most work.

Pick something a user would notice

An indicator should describe an experience, not an internal state. Successful requests over total requests. Requests served under some latency over total requests. Jobs completed within their window over jobs submitted.

The test is whether a customer could, in principle, tell the difference. Nobody has ever filed a ticket because your heap was fragmented. They file tickets because a page did not load.

Set the target from what you already deliver

The instinct is to pick a round, impressive number. Resist it. Look at what the service has actually done over the last quarter, and set the target slightly below that. If you have been running at 99.7 percent, do not promise 99.95 percent. You have no plan to get there, and a target you are permanently missing is worse than no target, because everyone learns to ignore it.

A target you are currently meeting is not a cop-out. Its job in the first six months is to detect regression, not to force improvement. Improvement comes later, once the number is trusted.

The budget is the part that changes behavior

The objective on its own is a dashboard. What makes it useful is the inverse: the amount of unreliability you have agreed is acceptable. That is a resource, and like any resource it can be spent.

When the budget is healthy, ship. Take the risk, deploy on a Friday if the pipeline is good, run the experiment. When it is exhausted, the team stops adding risk and spends the time on stability instead. The decision has been made in advance, in the calm, by everyone. Nobody has to win an argument at 2am.

What to expect early on

The first month is mostly discovering your instrumentation is wrong. Numbers will not match, health checks will turn out to be measuring the load balancer rather than the service, and something will be double-counted. This is normal and it is worth the trouble. You are not really running a service you cannot measure, you are hoping about it.

The second thing that happens is quieter. People start reaching for the number in conversations that used to be conducted in adjectives. That is the actual payoff.