LLM-as-judge grader for model outputs using the OpenAI API (Node.js)
A Node.js script that grades a JSONL file of model outputs against your written rubric with an OpenAI judge model, returning a 1-5 score and reasons per case, a pass rate, and CI-friendly exit codes.
Automation in Product by AISkillHub Library
What it does
String checks catch broken formats and missing facts, but not whether a reply got less clear, less correct in substance, or slipped in a promise it should not make. A model grading against an explicit rubric is a practical middle ground between regex checks and full human review, as long as you treat it as a noisy instrument.
**How it works**
1. You provide `OUTPUTS.jsonl`: one line per case with `id`, `input`, `output` and an optional `reference` (a known-good answer or facts). 2. You provide `RUBRIC.md`: criteria in priority order and what each score from 1 to 5 means. A sample support-reply rubric is included. 3. For each case the script calls `chat.completions.create` with JSON mode (`response_format: {type: "json_object"}`), asks for `{"score", "reasons"}`, and validates the reply. 4. Cases scoring below `--min-score` (default 4) fail. Unparseable judge replies are reported separately.
**Example output**
``` PASS refund-in-window: 5/5 Correctly states the 14-day window and the 5-10 day refund timing from the reference. FAIL password-reset: 1/5 Asks the user for their password, which the reference forbids, and does not address the missing email.
1/2 passed (min score 4), mean 3.00, judge <your judge model> ```
`--out graded.jsonl` saves each verdict with the judge model ID for later comparison.
**Use it well**
- Calibrate first: grade 20 to 30 cases yourself, run the judge, and adjust the rubric until you mostly agree. - Keep the judge model and rubric fixed when comparing two prompt versions; otherwise you are measuring the judge. - Avoid using the same model as both the system under test and the judge where you can. - The input and output are wrapped in tags and the judge is told to grade only, but a crafted output can still try to influence the judge. Do not use this as a security control.
**What it will not do**
It does not generate the outputs (run your system first), and it does not replace human review for launch decisions.
Exit codes: 0 all passed, 1 some below threshold, 2 config error, 3 API error, 4 judge returned invalid JSON for some case.
How to use it
1. Copy `scripts/llm-judge.mjs` and `evals/judge-rubric.md` into your repository. `evals/outputs.example.jsonl` shows the input format. 2. Install the SDK (Node.js 18+): `npm install openai` 3. Set environment variables (never commit the key): ``` export OPENAI_API_KEY=... export OPENAI_JUDGE_MODEL=... # a current OpenAI model ID that supports JSON mode ``` 4. Run: ``` node scripts/llm-judge.mjs evals/outputs.example.jsonl evals/judge-rubric.md node scripts/llm-judge.mjs outputs.jsonl evals/judge-rubric.md --min-score 4 --out graded.jsonl node scripts/llm-judge.mjs outputs.jsonl evals/judge-rubric.md --limit 10 # while tuning the rubric ``` 5. Rewrite `evals/judge-rubric.md` for your own task: list criteria in priority order and describe each score level concretely. 6. For CI, store `OPENAI_API_KEY` as a repository secret and `OPENAI_JUDGE_MODEL` as a variable, and run the script after the step that produces `outputs.jsonl`.
Files
Save each file at the path shown, relative to your project root.