A backup nobody has restored is a hypothesis

Every organisation that has lost data had backups. That is not a joke about negligence. It is the actual pattern. The backup job ran, the dashboard was green, and the restore failed or took four days. The failure is almost never in the taking; it is in the assumption that taking implies recovering.

Why the green tick lies

• The job succeeded but excluded a directory added last year, so the backup is complete and useless.
• The database was backed up while running, without a consistent snapshot, so the file restores and the database will not start.
• The encryption key lives on the server being backed up, and is therefore gone in exactly the scenario that requires it.
• The backup is on the same storage account as production, so a compromise or an accidental deletion takes both.
• Restore works, but takes 40 hours because nobody measured it against a business expectation of four.
• Retention is 14 days and the corruption started five weeks ago.

None of these are visible from a success rate. All of them are visible from a restore test.

Setting objectives that mean something

Two numbers per system, agreed with the people who own the business process rather than assumed by IT:

• RPO (recovery point objective): how much data you can afford to lose, measured in time. It sets backup frequency.
• RTO (recovery time objective): how long you can be without the system. It sets the recovery method, and it is usually the expensive one.

These differ enormously by system, and pretending otherwise is what makes disaster recovery unaffordable. An order database might justify an RPO of minutes; the internal wiki does not. Applying the strictest requirement uniformly is how DR budgets get rejected.

The 3-2-1 rule, and why it still holds

Three copies of the data, on two different media or platforms, with one off-site and ideally immutable. Ransomware changed the emphasis: the off-site copy now needs to be one that a compromised administrator account cannot delete. Object-lock or write-once storage is the practical form of that, and it is the single most valuable change most estates can make.

How to test without endangering production

1. 01Restore into an isolated environment, never over the live system.
2. 02Time it, from the decision to restore through to service available, including the parts people forget, like DNS and certificates.
3. 03Verify content, not just completion. Open the application, run a query, check a recent record exists. A restore that produces an empty database also 'succeeds'.
4. 04Have someone who did not build the system follow the written procedure, so you are testing the documentation at the same time.
5. 05Write down what went wrong. Something will. That record is the actual output of the exercise.

Quarterly is a reasonable cadence for most organisations, with the most critical system tested more often. The reporting matters as much as the test: measured recovery time against target, for each system, in a document you can hand to an auditor, an insurer or a board.

Backup is a cost. Recovery is the product. Only one of them has ever been tested in most organisations, and it is not the one that matters.

If you cannot currently say when your last successful restore test was, that is the finding. It is also the first thing we check when we take over an environment.