Postmortem template for AI and LLM feature incidents
A blameless incident postmortem template adapted for AI features: records model and prompt versions, user and data impact, agent actions taken, an AI-specific failure classification, layered root cause questions, and eval-focused action items.
File in Operations by AISkillHub Library
What it does
Standard postmortem templates miss what matters in an AI incident. Which model and prompt version produced the output? Did an agent take actions, and which? Was it injection, hallucination, retrieval or a provider change? Why did no eval cover it? This template asks those questions so the fix lands in the right layer and the failure becomes a permanent regression test.
**What is different from a generic template**
- **Header fields** for the model ID and prompt version (commit or tag) in use at the time, since both change independently of code deploys. - **Impact** includes data exposure status, actions taken by agents or tools, and cost impact from runaway loops. - **Failure classification** checklist: hallucination, instruction failure, prompt injection, data leakage, unsafe content, unintended action, format failure, model or provider change, retrieval failure, capacity or cost. - **Root cause prompts** that walk the layers: triggering input, why the model produced it, why guardrails did not stop it, why detection was slow, and why no eval caught it before release. - **Action items** typed as Prevent, Detect, Mitigate or Process, with the first row pre-filled: add the triggering inputs to the eval set. - **Eval and monitoring changes** section, including whether the kill switch was used and worked.
**Variables to fill**
`{{incident_title}}`, `{{incident_id}}`, `{{severity}}`, `{{detected_at}}`, `{{resolved_at}}`, `{{authors}}`, `{{feature_name}}`, `{{model_id}}`, `{{prompt_version}}`, `{{count_or_estimate_and_how_measured}}` and `{{follow_up_date}}`.
**What it will not do**
It does not replace your incident response process or severity scale; it plugs into them. For incidents involving personal data, it reminds you to involve legal or privacy but does not tell you what your notification obligations are.
Write it blameless: describe what the system and process allowed, not who made a mistake.
How to use it
1. Copy `docs/ai-incident-postmortem-template.md` into your incidents folder or wiki. 2. For each incident, duplicate the file and name it with the date and a short slug, e.g. `2026-03-14-refund-bot-injection.md`. 3. Fill the header table first while details are fresh, especially the model ID and prompt version that were live. 4. Replace each `{{...}}` field, tick all failure categories that apply, and work through the root cause questions in order. 5. Delete the block-quote guidance notes before sharing. 6. Create the action items in your tracker and link them in the table. Always complete action item 1: add the triggering inputs to your eval set. 7. Hold the follow-up review on `{{follow_up_date}}` to confirm the actions are done and the new eval cases pass.
Files
Save each file at the path shown, relative to your project root.