---
name: incident-debug
description: Structured, read-only investigation for an active production incident. Builds a timeline, links the symptom to recent code changes, traces errors into the code, ranks hypotheses with confirming checks, separates mitigation from fix, and keeps an incident log. Use when the user reports an outage, production errors, an alert, or says they are on call and something is broken.
---

# Incident debugging

Incident: `$ARGUMENTS`

## Ground rules

- Read-only. Do not run commands that change any system, deploy, restart, migrate, or delete. Local read-only commands (git, grep, reading files) are fine.
- When a check requires production access (logs, metrics, database queries), write the exact command or query for the user to run, and ask them to paste the result. Prefer read-only queries with a `LIMIT`.
- Separate facts (observed, with source) from inferences.
- Keep answers short. The user is under pressure.

## 1. Frame (ask only what is missing)

- Symptom, as the user or alert sees it.
- Start time with timezone. Is it ongoing?
- Scope: all users or some? One region, endpoint, tenant, platform?
- What does resolved look like?

## 2. Start the log

Create `incidents/<YYYY-MM-DD>-<slug>.md`:

```
# <title>
Status: investigating
## Timeline (UTC)
- HH:MM <event> (source)
## Facts
## Hypotheses
## Actions taken
## Open questions
```

Append to it after each step. Use UTC.

## 3. What changed

- `git log --since="<start minus 24h>" --until="<start plus 1h>" --oneline --merges` and the same without `--merges`.
- `git tag --sort=-creatordate | head` to map tags to deploys if the project tags releases.
- Ask the user about changes outside this repo: config, feature flags, infrastructure, dependencies, traffic, third-party status.
- For the most suspicious commits, read the diff.

## 4. Trace the error

- Search for the exact error message, error code, or top application frame from the stack trace.
- Read the code path that raises it. List the conditions that lead there.
- Note timeouts, retries, pool sizes and limits on that path; saturation often looks like random errors.

## 5. Hypotheses

Write two to five, ranked:

```
H1 (likely): <explanation>
  Fits: <evidence>
  Does not fit: <evidence, if any>
  Check: <command or query for the user, or a code read>
```

Update rankings as evidence arrives. Remove hypotheses that are ruled out and note why in the log.

## 6. Mitigation vs fix

- Mitigation options in order of risk: disable a feature flag, roll back the suspect deploy, scale or shed load, block a bad client. State expected effect and how to verify it.
- Root-cause fix: describe it and where it goes in code. Offer to write it only after mitigation is in place or the user asks.

## 7. Close

When the user says it is resolved, set `Status: resolved`, fill in the timeline, and add a short "Follow-ups" section (tests, alerts, runbook updates). Do not write blame or speculation into the log.