A backup you have never restored is not a backup. It is a file, and a belief about that file. The belief is usually wrong in some small way that only shows up when you need it.
This is one of the few areas of operations where the failure is almost entirely silent until the worst possible moment. A broken pipeline announces itself. A broken backup sits there, green, for years.
What actually goes wrong
The problem is rarely that the backup did not run. Monitoring usually catches that. The problems are subtler:
- The backup completed but was empty, or captured only part of the data, because a permission changed six months ago.
- The data is fine, but restoring it takes eleven hours and the recovery expectation everyone has in their head is one hour.
- The restore works, but produces a database with no users, no grants, or no extensions, and nobody knows the manual steps to finish it.
- The backup is encrypted with a key that lives in the system you are trying to recover.
That last one is not hypothetical thinking. Circular dependencies in recovery are common precisely because they are invisible when everything is running.
Restore drills
The fix is unglamorous: restore something, on a schedule, and time it. Not a checksum, not a "the file exists" check. An actual restore into an actual usable state, performed by a person following the written procedure.
Quarterly is enough for most systems. What you want out of it is three numbers: how long it took, how many steps were missing from the document, and whether the restored system passed a real check rather than just starting up.
The first drill is always embarrassing. That is the point. It is much better to be embarrassed on a Tuesday afternoon than during an actual data loss event.
Know your two numbers
Two figures should be written down for every important system, and agreed with whoever owns the business risk.
The first is how much data you can afford to lose, which sets your backup frequency. The second is how long you can afford to be down, which sets your restore approach. Hourly snapshots and a twelve-hour restore procedure are an incoherent pair, and it is worth noticing that on paper rather than in an incident.
Most teams have never had this conversation explicitly, and discover during the outage that the expectation was an order of magnitude tighter than the capability.
Test the boring failure, not the dramatic one
Disaster recovery planning tends to focus on the region going down. In practice the thing that happens is somebody runs a delete against the wrong environment, or a migration corrupts a column, or a bug quietly writes bad data for a week before anyone notices.
That last case is the one that catches people out, because it means the most recent backup is also bad. Point-in-time recovery matters more than most teams assume, and it is worth knowing whether you have it before you need it.