Twenty-one posts, one problem each
Short pieces that take a single operational problem and work it through. The longer positions live in the strategy and the playbooks; these are the specifics underneath them.
Keeping production up
Knowing something is wrong, finding out why, and getting the pager to a state where the people carrying it believe it.
Alerting
Alerts That Earn the Page
If it fires at 3am and there is nothing to do, it is not a page. How to build a rotation that believes its own pager.
Incidents
The First Hour of an Incident
Three roles and one shared document. What a small team actually needs, and why mitigation comes before understanding.
Incidents
Postmortems That Change Something
Most reviews produce a document and no change. The specific reasons why, and what to do differently.
SLOs
Your First SLO
Starting from one service and one number, when nobody has agreed that reliability should be measured yet.
Observability
Dashboards Nobody Opens
Ninety dashboards and four that get used. Building for a question instead of for a system.
Observability
Logs Are Not Free
Adding a log line is free when you write it and expensive at 3am. Levels, structure, cardinality, and retention.
Runbooks
The Runbook Test
A runbook is real if somebody else can run it on the worst night of the quarter without asking you anything.
On-call
On-Call Is a Design Constraint
The pager is a feedback loop, and where you point it decides what gets built.
Shipping changes safely
Everything between a commit being correct and a user experiencing it, which is where most of the risk actually lives.
Environments
Staging Lies to You
Staging answers a weaker question than the one you are asking. What it is genuinely good at, and what to do instead.
Standards
What Production-Ready Actually Means
Six questions specific enough that two people would reach the same verdict.
Delivery
Feature Flags Are an Operational Tool
Separating deploy from release is what makes frequent deployment safe. Also kill switches, load shedding, and flag debt.
Migrations
The Migration That Never Finishes
The most expensive state a system can be in is halfway between two designs. Plan the last service first.
Metrics
What Change Failure Rate Hides
The delivery metric most easily gamed without anyone intending to. What the definition is quietly doing.
Getting rid of toil
Repetitive manual work produces no lasting value and compounds quietly. What to attack first, and how.
Tech debt
Site Reliability Engineering: Your Secret Weapon Against Tech Debt
How SREs turn mounting technical debt into a manageable, scalable system.
Toil
The Five-Minute Manual Step
The task too small to justify a ticket and too frequent to ignore. Why the arithmetic is worse than it looks.
Automation
Automate the Thing You Hate
Not the highest-frequency task. The one you are dreading, because dread is a decent proxy for risk.
Maintenance
Dependency Upgrades Are Reliability Work
The least popular reliability work there is, and the one where the cost of skipping it is not linear.
Cost, capacity, and risk
The things that stay invisible until the moment they are extremely visible.
Backups
The Restore You Have Never Run
A backup you have never restored is a file and a belief about that file. The belief is usually wrong.
Capacity
Capacity Planning Without a Crystal Ball
You do not need to predict the future. You need headroom, growth rate, and how long it takes to add more.
Cost
Cost Is a Reliability Signal
A spend spike is often the earliest signal that something is wrong technically, and it beats your monitoring to it.
Security
Secrets, the Boring Way
The failures are never the cryptography. They are a credential in a place nobody thought of as a place.
The longer arguments
Each post here takes one problem. The strategy and the playbooks take the whole position, at length, and they get revised as I change my mind.