The person who set it up has left

It usually surfaces during an incident. Something breaks, and the answer to every question is that the person who built it does not work here any more. There is no diagram. The credentials are in a spreadsheet that may or may not be current. Nobody is confident enough to change anything, so the system is left alone, which is precisely how it becomes more fragile.

This is the most common condition we are called into, and it is worth being clear about what it costs before it fails: every change takes longer because it starts with archaeology; risk cannot be assessed, so it is either ignored or overestimated; and the organisation cannot switch providers, because nobody could hand over what they do not understand.

Discovery, in order

The instinct is to start fixing. Resist it. Changing an undocumented system is how a manageable situation becomes an outage. Map first.

1. 01Inventory. Every server, service, domain, certificate and third-party account. Include the things nobody mentions: the DNS registrar, the SSL provider, the monitoring account, the domain someone bought personally.
2. 02Ownership and billing. Which credit card renews each of those, and whose email receives the notice. Expired domains and lapsed certificates cause more outages than hardware does.
3. 03Dependencies. What talks to what, and which direction. This is the part that cannot be skipped, because it is what makes change safe.
4. 04Access. Who can reach each system today, including former staff and contractors whose accounts were never removed.
5. 05Data. What is stored where, what is backed up, and, separately, what has ever been restored.
6. 06Exposure. What is reachable from the public internet, deliberately or otherwise.

What actually belongs in a runbook

Most runbooks fail because they describe architecture instead of actions. Architecture belongs in the diagram. A runbook answers what to do at 3am, written for someone tired and under pressure:

• How to restart each service safely, including the order and anything that must be stopped first
• How to restore from backup, with the actual commands and the expected duration
• How to fail over, and just as important, how to fail back
• Where the logs are and which ones matter for which symptom
• Who to escalate to, with a second name in case the first is unreachable
• What normal looks like, so abnormal is recognisable

A runbook nobody has followed is a draft. Walk through each procedure once, with someone who did not write it, and correct what turns out to be wrong. It always is.

Preventing the recurrence

Documentation written as a project decays, because it is separate from the work. Documentation produced as a by-product of the work survives. Three practices carry most of the weight:

• Define infrastructure as code. A Terraform repository is documentation that cannot drift from reality, because it is the thing that creates reality.
• Put credentials in a shared vault with named access, never in a spreadsheet, a wiki page or a person's password manager.
• Require that every change leave a record: what changed, why, who approved it, how to reverse it. This is also, conveniently, most of what an auditor will later ask for.

Key-person risk is not solved by asking the key person to write more things down. It is solved by making the system describe itself.

If you have inherited an estate like this, the discovery phase is the deliverable that matters most, and it is worth having in writing even if you then run the environment yourself.