The fastest way to make an on-call rotation miserable is to page people for things they cannot act on. It happens gradually. Someone adds an alert after an incident because an alert feels like a fix. Nobody removes it when the underlying cause is dealt with. Two years later the rotation is receiving forty pages a week, thirty-eight of them are noise, and the two that mattered arrive looking exactly like the rest.
The test I apply to any alert is short: if this fires at 3am, is there something a human should do right now? If the answer is no, it is not a page. It might still be a ticket, a dashboard panel, or a weekly report, but it does not get to wake somebody.
Symptoms page, causes inform
The most common failure is alerting on causes. CPU is high. Disk is at eighty percent. A pod restarted. None of these are inherently bad. A service can run at ninety percent CPU perfectly happily, and a pod restarting is sometimes the system healing itself exactly as designed.
Alert on the thing a user would notice instead: requests are failing, latency has crossed the threshold you promised, a queue is growing faster than it drains. Those are finite and stable. The list of causes is neither. You will never finish enumerating the ways a system can break, but you can enumerate the ways it can be broken from outside.
Causes are still worth collecting. They belong in the dashboard you open after the page, where they help you find the problem quickly. The difference is that they inform the response rather than trigger it.
Every page needs an owner and a next step
An alert should arrive with three things attached: what broke in user terms, where to look, and what to do first. If the runbook link goes to a page that says "investigate the issue," the alert is not finished. That is not documentation, it is a shrug with a URL.
This is also a good forcing function. If you cannot write down what the responder should do, you probably do not understand the failure well enough to be alerting on it yet.
Deleting alerts is real work
Teams add alerts far more readily than they remove them, because adding one feels responsible and removing one feels like accepting risk. It is worth being deliberate about this. Once a quarter, pull the list of everything that paged, sort by frequency, and look at the top ten. For each one ask what the responder actually did. If the honest answer is "acknowledged it and went back to sleep," you have found something to delete or downgrade.
The reason to bother is that alert fatigue is not a morale problem, it is a reliability problem. A rotation that has learned to ignore its pager will ignore the real one too, and the cost of that lands during the incident you actually needed them awake for.
The quiet measure of a good alerting setup
A healthy setup is boring. The pager is quiet most weeks. When it does fire, the responder believes it immediately, because it has earned that. Getting there is mostly subtraction: fewer alerts, tied to things users feel, each with a next step written down.