Human review rubric and scoring sheet for RAG answers
A four-dimension rubric (grounding, citation, answer handling, usefulness) with a strict pass rule and a CSV scoring sheet, so reviewers grade retrieval-augmented answers consistently and separate search failures from generation failures.
File in Product by AISkillHub Library
What it does
Reviewing RAG answers by gut feel produces scores nobody can compare across weeks. The common failure is also subtle: an answer that happens to be right but was not supported by the retrieved passages. That system will be confidently wrong next time. This rubric makes reviewers check what matters, in a fixed way.
**The four dimensions** (each scored 0, 1 or 2)
- **Grounding**: is every claim that matters supported by the retrieved passages? Unsupported numbers, dates, prices and policy terms always score 0. - **Citation**: do citations point to the passage that actually supports each claim? Marked `na` if your system does not cite. - **Answer handling**: did the system answer when the passages contained the answer, and say "I don't know" plainly when they did not? - **Usefulness**: does it answer the question first and give what the user needs to act?
**Pass rule**: Grounding 2, Answer handling 2, Usefulness at least 1, Citation 2 or `na`. Everything else fails.
**Retrieval miss** is tagged separately: the right answer exists in your corpus but was not retrieved. Tracking it apart tells you whether to work on search or on the prompt.
**Files**
- `evals/rag-answer-rubric.md`: the rubric with score tables, the pass rule and reviewer notes (calibrate on 20 shared items first, do not change the rubric mid-round, write one specific sentence for every 0). - `evals/rag-review-sheet.csv`: a scoring sheet with one row per answer and two worked examples, ready to open in any spreadsheet tool.
**What it will not do**
It does not grade automatically. You can adapt the rubric text for an LLM judge, but calibrate that judge against human scores from this sheet first. It also assumes reviewers can see the exact retrieved passages; if your system does not log them, fix that before reviewing.
How to use it
1. Copy `evals/rag-answer-rubric.md` and `evals/rag-review-sheet.csv` into your repository or shared drive. 2. Export a sample of RAG interactions with three things per row: question, retrieved passages as the model saw them, and the answer. 3. Open the CSV in a spreadsheet, delete the two example rows, and add one row per answer. Use `case_id` to link back to the logged interaction. 4. Calibrate: two reviewers score the same 20 answers, then discuss every disagreement on grounding and answer handling. 5. Split the remaining answers and score them. Write a specific note for every 0. 6. Report pass rate and retrieval-miss rate separately. Re-use the same rubric version for the next round so numbers are comparable.
Files
Save each file at the path shown, relative to your project root.