Lorixen.
EN

Blameless postmortems for a small web team

Short answer: When something goes wrong with your website or app, such as an outage, lost orders or a botched deploy, the most valuable thing you can do afterwards is write a short postmortem: what happened, when, what the impact was, why it happened and what you will change. Keep it blameless, meaning it looks for causes in systems and processes, not in the person who pressed the button. Agree in advance what kind of incident deserves one, give every action item an owner and a date, and keep the documents somewhere everyone can find them.

Why small teams skip postmortems, and why they shouldn't

In a small business the same two or three people fix every problem, and once the site is back up everyone is relieved and returns to their real work. A week later nobody remembers the exact sequence of events, and three months later the same thing happens again.

A postmortem is the cheap insurance against that. It does not need a meeting room or a template library. For a small team it can be a one- or two-page document, written within a few days while memories are fresh, and read by everyone who touches the site or app.

Decide in advance when to write one

Google's Site Reliability Engineering book, in its chapter on postmortem culture, recommends setting criteria before an incident happens so that everyone knows when a postmortem is necessary. Its list of common triggers includes:

  • User-visible downtime or degradation beyond a threshold.
  • Data loss of any kind.
  • On-call engineer intervention, such as a release rollback or rerouting traffic.
  • A resolution time above a threshold.
  • A monitoring failure, which usually implies the problem was discovered manually.

The chapter adds that any stakeholder may request a postmortem for an event. For a small business, a practical version might be: the site or checkout was unavailable for more than 15 minutes, any order or customer data was lost or wrong, a customer told us before our monitoring did, or anyone asks for one.

What "blameless" means

The same chapter calls blameless postmortems a tenet of SRE culture. It says a truly blameless postmortem focuses on identifying the contributing causes of an incident without indicting any individual or team for bad or inappropriate behaviour, and assumes everyone involved had good intentions and did the right thing with the information they had.

This matters even more in a small team. If the person who deployed the broken change expects to be blamed, the next incident report will be vague and defensive. If they expect the team to ask "what made that mistake easy to make?", you get the full story, including the near-misses nobody else saw. Atlassian's guide to blameless postmortems makes a similar point: the goal is to examine honestly and objectively the circumstances that led to the fault so the true causes can be found and mitigated.

A simple template

Use the same headings every time so postmortems are quick to write and easy to compare:

  1. Summary. Two or three sentences a non-technical owner can understand.
  2. Impact. Who was affected, for how long and how: failed checkouts, wrong prices, emails not sent.
  3. Timeline. Times of key events: when it started, when it was noticed and by whom, each action taken, when it was resolved.
  4. Contributing causes. Usually several. A bad deploy is one cause; missing tests, no rollback plan and no alert are others.
  5. What went well. Quick detection, a clear rollback, good customer communication.
  6. Action items. Specific, owned and dated.
  7. Lessons. Anything worth sharing beyond the immediate fixes.

Getting to the causes

Ask "why" repeatedly, but stop when you reach something you can change. "The developer forgot to run the migration" leads to "why was it possible to deploy without the migration?", which leads to an action: run migrations automatically in the deploy pipeline.

Look at four areas: the change that triggered the incident, how it got through review and testing, how quickly it was detected, and how quickly it was fixed. Detection and recovery improvements are often cheaper than preventing every possible failure.

Action items that actually get done

The most common way postmortems fail is a list of good intentions that nobody does. Make each item:

  • Specific: "Add an alert when the checkout error rate exceeds 2% for five minutes", not "improve monitoring".
  • Owned: one named person.
  • Dated: with a realistic deadline.
  • Tracked: in the same tool you use for other work, not just in the document.

Review open items at a regular team check-in until they are closed.

Sharing and storing

The SRE chapter notes that once those involved are happy with the document and its action items, it is added to a repository of past incidents, and that transparent sharing makes it easier for others to find and learn from it. A shared folder or wiki page with one document per incident is enough. Before a big change, such as a platform migration, read through the past ones.

A worked example

A hypothetical Wellington meal-kit company had a two-hour outage on a Sunday evening, its busiest ordering window, when a payment plugin update conflicted with the checkout theme. A customer emailed before anyone on the team noticed.

The postmortem, written on Tuesday, listed four contributing causes: plugin updates applied automatically to the live site, no staging test of checkout, no alert on failed payments and no documented rollback. Action items included turning off automatic plugin updates, adding a checkout test on staging, creating a failed-payment alert and writing a one-page rollback guide. All four were done within three weeks, and the next problematic plugin update was caught on staging.

FAQ

Should customers see our postmortems?

Internal documents can stay internal, but a short, honest public summary after a major incident often builds trust.

Who should write it?

Usually the person who led the response, with input from everyone involved. Rotating the role spreads the skill.

How long should it take?

For a small incident, an hour or two to write and a short review. Larger incidents deserve more time.

If the same kinds of outages keep coming back to your site or app, tell us about the last one.

Sources

WhatsAppBook a call