Prompt regression runner for the Anthropic API with a PR workflow
A Python script that runs JSONL test cases through the Anthropic Messages API and checks each reply with contains, regex, length and JSON rules, plus a GitHub Actions workflow that runs it on PRs touching prompts.
Automation in Engineering by AISkillHub Library
What it does
When you edit a system prompt or switch models, you need to know whether the answers you rely on still come back right. This runner gives you a fast, cheap regression check: a JSONL file of cases, deterministic checks on each reply, and a non-zero exit code when anything regresses, so CI can block the merge.
**How a case looks**
```json {"id": "triage-json", "system": "Classify the message. Reply with only JSON with keys category and urgent.", "input": "I was charged twice and need it fixed today.", "max_tokens": 200, "checks": {"json": true, "json_keys": ["category", "urgent"], "contains": ["billing"]}} ```
Use `system_file` instead of `system` to point at a prompt file in your repo, so the eval tests the exact text you ship.
**Available checks**: `contains` and `not_contains` (case-insensitive), `regex`, `max_chars`, `json` (reply must parse, code fences stripped) and `json_keys`. A reply cut off at `max_tokens` is always reported as a failure.
**What you get**
``` PASS refund-window (1.8s, 62 in / 71 out tokens) FAIL triage-json (0.9s, 58 in / 40 out tokens) - JSON missing keys: urgent 2/3 passed on <your model ID> ```
`--out results.jsonl` saves every reply with its token usage, and the workflow uploads that file as an artifact.
**Design choices**
- The model ID comes only from `ANTHROPIC_MODEL`. There is no default, so a model change is always a visible config change. - Uses the official `anthropic` SDK with `max_retries=3`. The API key is read from `ANTHROPIC_API_KEY` by the SDK. - On forked PRs, where secrets are unavailable, the workflow skips with a notice instead of failing.
**What it will not do**
These are string and structure checks, not quality grading. They catch regressions like a missing policy detail or broken JSON, not a reply that became vaguer. Model output varies between runs, so write checks against what must always be true, and pair this with an LLM-judge or human review for tone and helpfulness.
Exit codes: 0 all passed, 1 a check failed, 2 configuration error, 3 API error.
How to use it
1. Copy `scripts/prompt_regress.py`, `evals/cases.jsonl` and `.github/workflows/prompt-regression.yml` into your repository. 2. Replace the starter cases in `evals/cases.jsonl` with cases for your prompts, one JSON object per line with a unique `id`. 3. Run locally: ``` pip install anthropic export ANTHROPIC_API_KEY=... # from the Anthropic Console; never commit it export ANTHROPIC_MODEL=... # a current model ID from Anthropic's models page python3 scripts/prompt_regress.py evals/cases.jsonl --out results.jsonl python3 scripts/prompt_regress.py evals/cases.jsonl --only triage-json ``` 4. In GitHub, go to Settings > Secrets and variables > Actions: - add a **secret** `ANTHROPIC_API_KEY` - add a **variable** `ANTHROPIC_MODEL` with the model ID to test 5. Edit the `paths:` filter in the workflow if your prompts live somewhere other than `prompts/`.
Each case makes one API call, so cost scales with the number of cases. Keep the CI set focused and run larger sets on demand with `workflow_dispatch`.
Files
Save each file at the path shown, relative to your project root.