Every team I have worked with has had a staging environment, and every team has had at least one incident that staging failed to catch while everyone assumed it would. This is not a staging problem exactly. It is a problem with what we expect staging to be.
The ways it differs
Staging usually differs from production in ways nobody has written down. Less data, and different data. Fewer instances. A smaller database with a different query plan because the optimizer sees a different table size. No real traffic pattern, so no cache behaviour that resembles the real thing. Different credentials, sometimes different permissions entirely. Often a different scaling configuration, because running production-sized infrastructure twice is expensive.
Each of those is individually reasonable. Together they mean staging is answering a different question than the one you are asking. "Does this work in staging" is not a prediction about production, it is a much weaker claim: this is not obviously broken.
What staging is actually good at
It is worth being clear that staging is useful, just narrower than we pretend. It reliably catches things that are broken everywhere: a bad migration, a missing environment variable, a service that will not start, an integration whose contract you got wrong. Those are common and cheap to find there.
What it does not catch is anything that depends on scale, on real data shapes, on concurrency, or on the specific state production has accumulated over three years. Which is most of the interesting failures.
Stop trying to make it identical
The tempting response is to make staging more production-like. Some of that is worth doing, particularly the parts that are free: same deployment mechanism, same configuration shape, same container image. But chasing full parity gets expensive quickly and never quite arrives, and the closer you get the more you trust it, which is its own hazard.
The better investment is usually to get faster at finding out in production, safely. That means the release mechanics: deploying without releasing, shifting a small percentage of traffic first, watching the right signals for a few minutes, and being able to reverse quickly when the answer is bad.
Design for the difference
A few habits help:
- Write down how the two environments differ. The undocumented differences are the ones that hurt.
- Treat a staging pass as a gate, not a guarantee. It means "no known problems," not "safe."
- Put the same instrumentation in both. If you cannot see it in staging, you are also blind for the first ten minutes in production.
- Make the rollback path the well-tested one, since it is the thing you will actually depend on when staging was wrong.
None of this means deleting staging. It means holding it to an honest job description. It is a cheap filter for obvious breakage, and the last thing that will tell you whether something works.