Skills

Flaky test triage: reproduce, classify the cause, propose a fix

A Claude Code skill that takes one intermittently failing test, reruns it to measure the failure rate, classifies the likely cause (timing, shared state, clock, randomness, network, ordering) and proposes a targeted fix.

Skill in Engineering by AISkillHub Library

What it does

A flaky test is a test that passes and fails on the same code. The usual responses, adding retries or bumping a timeout, hide the problem. This skill gives Claude Code a disciplined process for one flaky test at a time.

**What it does** 1. **Reproduce.** Runs the single test repeatedly in a loop and records pass/fail counts. Then runs it together with the rest of its file or package to check for order dependence. 2. **Classify.** Reads the test and the code under test and checks each known flakiness cause: async work not awaited, fixed sleeps, shared mutable state between tests, leftover database rows, real clock or timezone use, unseeded randomness, real network calls, port or file collisions under parallel runs, and unordered collections compared as ordered. 3. **Fix.** Proposes the smallest change that removes the cause, such as awaiting a condition instead of sleeping, faking the clock, isolating state per test, or sorting before comparing. 4. **Verify.** Reruns the loop after the fix and reports the new pass count.

**Output** A short triage note: failure rate before and after, the cause with the evidence (file and line), the fix, and whether the test should be quarantined while the fix is reviewed.

**What it will not do** It will not add automatic retries or increase timeouts as the fix unless it can show the operation is legitimately slow. If it cannot reproduce the failure locally, it says so and lists what CI-only factors to check (parallelism, slower machines, timezone, environment variables) instead of guessing.

How to use it

1. Save the file below as `.claude/skills/flaky-test-triage/SKILL.md` in your repository. 2. In Claude Code run `/flaky-test-triage <test id>`, for example `/flaky-test-triage tests/test_orders.py::test_refund_total` or `/flaky-test-triage "OrderService refunds partial payments"`. 3. Paste the CI failure output into the conversation if you have it; it speeds up classification. 4. The default loop is 20 runs. Ask for more ("run it 100 times") for rare failures. Long suites may take a while.

Files

Save each file at the path shown, relative to your project root.

  • .claude/skills/flaky-test-triage/SKILL.md3.6 KB

More skills

See all
  • Skill· EngineeringFree

    One-at-a-time dependency upgrade skill with changelog review

    A Claude Code skill that upgrades a single dependency (or a short list, one by one), reads its release notes for breaking changes, updates affected call sites, and verifies with your typecheck, tests and build.

    Coding agentsClaude

    ALAISkillHub Library

  • Skill· WritingFree

    /release-notes: user-facing release notes between two git refs

    A Claude Code slash command that turns the changes between two tags or commits into release notes written for end users and admins: highlights, improvements, fixes, and any action they must take to upgrade.

    Coding agentsClaude

    ALAISkillHub Library

  • Skill· EngineeringFree

    Incident debugging runbook skill: evidence first, read-only

    A Claude Code skill for active production incidents that builds a timeline, ties symptoms to recent changes in the repo, ranks hypotheses, and suggests read-only checks, while keeping a running incident log you can paste into Slack or a postmortem.

    Coding agentsClaude

    ALAISkillHub Library

Get AISkillHub updates in your inbox.

New features as they ship, AI live chat improvements, and the best new listings in the catalog.

Product updates, new features, and guides. Unsubscribe any time.