Flaky test triage: reproduce, classify the cause, propose a fix
A Claude Code skill that takes one intermittently failing test, reruns it to measure the failure rate, classifies the likely cause (timing, shared state, clock, randomness, network, ordering) and proposes a targeted fix.
Skill in Engineering by AISkillHub Library
What it does
A flaky test is a test that passes and fails on the same code. The usual responses, adding retries or bumping a timeout, hide the problem. This skill gives Claude Code a disciplined process for one flaky test at a time.
**What it does** 1. **Reproduce.** Runs the single test repeatedly in a loop and records pass/fail counts. Then runs it together with the rest of its file or package to check for order dependence. 2. **Classify.** Reads the test and the code under test and checks each known flakiness cause: async work not awaited, fixed sleeps, shared mutable state between tests, leftover database rows, real clock or timezone use, unseeded randomness, real network calls, port or file collisions under parallel runs, and unordered collections compared as ordered. 3. **Fix.** Proposes the smallest change that removes the cause, such as awaiting a condition instead of sleeping, faking the clock, isolating state per test, or sorting before comparing. 4. **Verify.** Reruns the loop after the fix and reports the new pass count.
**Output** A short triage note: failure rate before and after, the cause with the evidence (file and line), the fix, and whether the test should be quarantined while the fix is reviewed.
**What it will not do** It will not add automatic retries or increase timeouts as the fix unless it can show the operation is legitimately slow. If it cannot reproduce the failure locally, it says so and lists what CI-only factors to check (parallelism, slower machines, timezone, environment variables) instead of guessing.
How to use it
1. Save the file below as `.claude/skills/flaky-test-triage/SKILL.md` in your repository. 2. In Claude Code run `/flaky-test-triage <test id>`, for example `/flaky-test-triage tests/test_orders.py::test_refund_total` or `/flaky-test-triage "OrderService refunds partial payments"`. 3. Paste the CI failure output into the conversation if you have it; it speeds up classification. 4. The default loop is 20 runs. Ask for more ("run it 100 times") for rare failures. Long suites may take a while.
Files
Save each file at the path shown, relative to your project root.