An incident postmortem is a short written review of an outage: what happened, how it affected users, why it happened and what you will change so it does not happen again. A blameless postmortem focuses on systems and processes rather than on who made a mistake. The basic template has six parts: summary, impact, timeline, root cause, what went well and badly, and action items with owners and dates.
Below you will find when to write one, the principles that keep it useful, a full template to copy, a filled-in example and how to turn it into a customer-facing summary.
When to write a postmortem
Not every blip needs a document. Set clear triggers so the decision is not made under pressure. Common ones:
- Customers could not use a core function, such as checkout, login or the API, for more than a few minutes.
- Data was lost, corrupted or exposed.
- The incident used a large share of your monthly error budget.
- Customers or a client reported the problem before monitoring did.
- The same kind of incident has happened before.
Write it within a few working days, while memories and logs are fresh. A postmortem written three weeks later is mostly guesswork.
Blameless principles
A blameless postmortem assumes that everyone acted reasonably with the information they had at the time. If a person could take the site down with one command, the problem is that one command could take the site down. Blame makes people hide details, and the details are what you need.
- Describe actions, not people. “The import job was started during peak hours” instead of “Anna started the import at the wrong time”.
- Ask “how”, not “who”. How did the system allow this? How did we find out? How could we have found out sooner?
- Look for several causes. Incidents rarely have one root cause. Usually a trigger meets a weakness that a missing safeguard did not catch.
- Use facts and timestamps. Monitoring data, logs and chat history beat memory.
- Keep the action list short. Two or three changes that get done beat ten that do not.
The postmortem template
Copy this into your wiki, repository or document tool. Each heading has a prompt for what belongs there.
# Postmortem: [short title]
Date of incident: [YYYY-MM-DD]
Author(s): [names]
Status: [draft / in review / final]
Severity: [e.g. SEV1 / SEV2 / SEV3]
## Summary
[Two or three sentences: what broke, for how long, and how it was fixed.]
## Impact
- Affected services and components:
- Duration (start to recovery):
- Users or customers affected:
- Business impact (orders, signups, SLA):
- Customer communication (status page, emails, tickets):
## Detection
- How was the incident detected? (monitor, customer, team member)
- Time from start to alert:
- Time from alert to acknowledgement:
## Timeline
[All times in one time zone. One line per event.]
HH:MM event
HH:MM event
## Root cause and contributing factors
- Trigger:
- Underlying cause:
- Why safeguards did not catch it:
## Resolution
[What was done to restore service, and by whom (role, not blame).]
## What went well
-
## What went badly
-
## Where we got lucky
-
## Action items
| Action | Type (prevent / detect / respond) | Owner | Due date | Ticket |
|--------|-----------------------------------|-------|----------|--------|
| | | | | |
## Customer-facing summary
[Short version for the status page or client email.]
The “Where we got lucky” section is easy to skip and often the most valuable. It captures the near misses: the incident happened at night, the backup had run an hour earlier, the one person who knew the system happened to be awake.
A filled-in example
Here is the template applied to a fictional incident at an online shop, northwind.shop. The details are invented for illustration, but the structure is what a real postmortem should look like.
Summary
From 03:12 to 03:19 CET, the checkout at northwind.shop returned 503 Service Unavailable. A nightly product import used all PHP-FPM workers, so checkout requests were rejected. The on-call developer stopped the import and restarted PHP-FPM. Total downtime: 7 minutes.
Impact
- Checkout and cart were unavailable for 7 minutes. Product pages were served from cache and stayed up.
- Server logs show 23 failed checkout requests in the window (example figure).
- The status page showed “Checkout: major outage” from 03:13. No support tickets were received.
Timeline
- 03:00 Nightly product import starts.
- 03:12 The uptime check to northwind.shop/checkout returns 503.
- 03:13 A second check confirms the failure. An incident opens, the on-call developer is alerted on Telegram and details are posted in the team’s Slack channel. The status page marks Checkout as down.
- 03:15 On-call acknowledges in Slack and opens the server dashboard. Memory and CPU are high on web-01.
- 03:17 PHP-FPM status shows all workers busy with import requests. The import job is stopped.
- 03:18 PHP-FPM is restarted.
- 03:19 Checks pass again. A recovery notice goes out with a total downtime of 7 minutes.
Root cause and contributing factors
- Trigger: the supplier feed was three times larger than usual after a catalogue update.
- Underlying cause: the import runs through the same PHP-FPM pool as the shop. Large imports can use every worker.
- Missing safeguard: the import has no limit on concurrent requests, and there was no alert on worker saturation.
For background on why exhausted workers show up as this particular error, see our guide to the 503 Service Unavailable error.
What went well, badly and lucky
- Well: detection took one minute and acknowledgement two. The status page updated without manual work.
- Badly: the cause took four minutes to find because nothing linked the import job to the web servers.
- Lucky: it happened at 03:12, not during the evening peak.
Action items
| Action | Type | Owner | Due |
|---|---|---|---|
| Run the import in a separate worker pool with a concurrency limit | Prevent | Backend lead | Within 2 weeks |
| Alert when web-01 memory stays above 85% | Detect | Ops | This week |
| Add a heartbeat that reports when the import finishes | Detect | Backend lead | This week |
| Add the “503 on checkout” runbook to the monitor | Respond | On-call rotation | This week |
Publishing a customer-facing summary
The internal postmortem is for the team. Customers need a shorter version without server names: what happened, how long, whether their data is safe and what you are changing. Post it on your status page and link to it from any incident email or client update.
Checkout unavailable, 03:12 to 03:19 CET
Between 03:12 and 03:19 CET, customers could not complete purchases on
northwind.shop. Browsing products was not affected, and no orders or
payment details were lost.
The cause was a scheduled product update that used more server capacity
than expected. We stopped it, and checkout recovered at 03:19.
We are moving these updates to separate capacity so they cannot affect
checkout again, and have added earlier warnings for this type of load.
Keep the tone factual and the promises concrete. “We take reliability seriously” says nothing; “imports now run on separate capacity” says something. Our guide on how to create a status page covers wording for incident updates in more detail.
It also helps to know how many people noticed. Uptimecheck includes status page analytics with every status page, so you can add to the postmortem how many visits the page had during the incident and from which countries. Combined with the monitoring timeline from uptime monitoring, that gives you start, alert, recovery and reach without digging through logs.
Tracking action items
A postmortem without finished action items is a story, not a fix. Treat the actions like any other work:
- Create a ticket for every action in the same tracker the team already uses, and link it from the postmortem.
- Give each one a single owner and a due date.
- Label actions by type: prevent, detect or respond. If every action is “be more careful”, the review was not finished.
- Review open postmortem actions in a regular meeting, for example every two weeks.
- Mark the postmortem as final only when all actions are done or explicitly dropped with a reason.
Over time, the postmortems themselves become data. Count how many incidents were found by monitoring versus customers, and track detection and response times from one incident to the next. Our article on MTTR, MTTA and MTBF explains how to calculate them.
Running the review meeting
For larger incidents, a 30 to 45 minute meeting helps. Share the draft beforehand, let the person who led the response walk through the timeline, and spend most of the time on contributing factors and actions. A neutral facilitator who did not work on the incident keeps the conversation blameless and on time. End by confirming owners and dates for each action item.
Get the timestamps for your next postmortem automatically. Try Uptimecheck free for 14 days, no credit card required.