Incident runbooks: fix outages faster when it matters
Short answer: A runbook is a short, practical guide for handling one specific problem, such as "checkout errors", "site down" or "emails not sending", written in advance so whoever is on duty can act quickly at 3 a.m. without guessing. Good runbooks say what the alert means, how to confirm the problem, the first safe steps to reduce harm, how to fix common causes, when to escalate and who to tell. Keep them close to the alert that triggers them, short enough to follow under stress, and updated every time they are used.
Why runbooks make such a difference
When something breaks, the person responding is often not the person who built that part of the system. They are tired, under pressure and possibly new. Without guidance they search old chat messages, guess, or wake up the one person who knows.
Google's Site Reliability Engineering book is specific about the value of preparation. Its introduction says that, in Google's experience, thinking through and recording best practices ahead of time in a playbook produced roughly a threefold improvement in mean time to repair compared with "winging it", and that the practiced on-call engineer armed with a playbook works much better than the hero who knows everything.
For a small business, that is the difference between a checkout outage lasting fifteen minutes or most of an evening.
What a good runbook contains
Google's SRE workbook describes playbooks as containing high-level instructions on how to respond to automated alerts: the severity and impact of the alert, debugging suggestions and possible actions to mitigate impact and fully resolve it. A practical template for each runbook:
- Title and trigger. Which alert or symptom this covers, such as "Checkout error rate high".
- What it means for customers. For example, "Customers cannot complete payment; orders are being lost."
- How to confirm. Where to look: a dashboard link, a test purchase with a test card, the payment provider's status page.
- First safe actions. Steps that reduce harm quickly without making things worse, such as rolling back the last deploy, switching on a maintenance banner or pausing a marketing email that is driving traffic.
- Common causes and fixes. The two or three usual culprits, each with exact steps.
- When to escalate. Time limits and conditions, with names and contact details.
- Who to inform. Support staff, the business owner, customers through a status page or social media.
- After the incident. Where to record what happened and whether a postmortem is needed.
Write for the person under pressure
- Short sentences and numbered steps. No essays.
- Links, not descriptions. "Open the error dashboard" with the link is better than "check the monitoring tool".
- Exact commands or button names, copied from a real successful use.
- Say what is safe. Mark steps that cannot be undone, such as deleting data or refunding orders, and say who may approve them.
- Assume no context. Write as if for a capable colleague who has never seen this part of the system.
Which runbooks to write first
Start with the problems that cost the most or happen most often:
- Website or app completely unavailable.
- Checkout or payment failures.
- Orders or bookings not reaching staff or the warehouse.
- Login failures for customers.
- Transactional emails or texts not sending.
- A suspected security incident, such as a leaked key or hacked admin account.
- Restoring from backup.
Five or six good runbooks cover most real incidents for a typical small business.
Keep them where people will find them
A runbook nobody can find in the moment does not exist. Link each runbook directly from the alert that triggers it, so the notification itself contains the link. Keep all runbooks in one place, such as a shared wiki or a folder in your code repository, and make sure it is reachable even when your main systems are down: a wiki hosted on the same server as your website is not much use when that server fails.
Keep them current
The SRE workbook recommends making sure playbooks are up to date with advice for each alert type, and having on-call engineers update the playbook with fresh information when the corresponding page fires. Practical habits:
- After every incident, the responder spends ten minutes updating the runbook used.
- Review all runbooks every few months, or after major changes such as a new hosting provider.
- Record the date of the last review at the top.
- Delete runbooks for systems you no longer have.
Practise
The workbook also suggests practising emergency response; it describes "Wheel of Misfortune" exercises, where team members role-play past incidents. A small team can do a light version: once a quarter, pick a runbook, walk through it on a test environment and fix anything that is unclear or outdated. New team members learn the systems, and the runbook improves.
A worked example
Picture a hypothetical Seattle meal-kit company whose two developers took turns on call. When the order export to the packing warehouse failed one Sunday, the developer on duty spent hours working out where the export ran and how to resend it.
The team wrote six runbooks using a shared template, starting with order export failures, checkout errors and site outages, each linked from its alert. The order export runbook lists how to confirm the failure, how to resend a day's orders safely, who to call at the warehouse and when to wake the second developer. The next export failure was fixed in under twenty minutes by the newest team member, who then added a missing step to the runbook.
FAQ
Isn't automation better than runbooks?
Automate repetitive fixes where you can. Runbooks cover what still needs judgement, and often lead to automation later.
Who should write them?
Whoever knows the system best, then someone else should test them.
How long should a runbook be?
Usually one screen. Link to detailed documentation rather than including it.
If only one person on your team knows how to fix things when they break, tell us which problems worry you most.