# RAG answer review rubric

Use this rubric when a person reviews answers from a retrieval-augmented (RAG) system: the model receives retrieved passages and must answer a question from them. Score each answer on the four dimensions below, then apply the overall rule. Record scores in `rag-review-sheet.csv`.

Reviewers need three things side by side: the question, the retrieved passages exactly as the model saw them, and the answer. Do not score from memory of the source documents. If the answer is right but the passages did not support it, that is a grounding failure, because next time the model will guess wrong just as confidently.

## 1. Grounding (G)

Is every factual claim in the answer supported by the retrieved passages?

| Score | Meaning |
|---|---|
| 2 | Every claim is supported by a passage. Reasonable paraphrase is fine. |
| 1 | Main claim supported; one minor detail is unsupported but harmless (e.g. a generic tip). |
| 0 | Any claim that matters is unsupported or contradicts the passages. |

Unsupported numbers, dates, prices, names, version numbers and policy terms always score 0.

## 2. Citation (C)

Skip (write `na`) if your system does not ask for citations.

| Score | Meaning |
|---|---|
| 2 | Each claim points to the passage that supports it. |
| 1 | Citations present but at least one points to the wrong passage or is missing for a key claim. |
| 0 | No citations, or citations to passages that do not contain the claim. |

## 3. Answer handling (A)

Did the system do the right thing given what was retrieved?

| Score | Meaning |
|---|---|
| 2 | Passages contained the answer and the system answered it; or they did not, and the system said so plainly without guessing. |
| 1 | Answered, but hedged so much the user cannot act; or declined when a partial answer was available. |
| 0 | Guessed when the passages lacked the answer, or declined when the answer was clearly present. |

## 4. Usefulness (U)

| Score | Meaning |
|---|---|
| 2 | Directly answers the question first, then needed detail. A user could act on it. |
| 1 | Correct but buried, padded, or missing a step the user will obviously need. |
| 0 | Does not address the question that was asked. |

## Overall rule

- **Pass**: G = 2, A = 2, and U >= 1, and C is 2 or `na`.
- **Fail**: anything else.
- **Retrieval miss** (tag it separately in the `retrieval_miss` column): the correct answer exists in your corpus but not in the retrieved passages. This is a search problem, not a generation problem; track it apart so you fix the right component.

## Reviewer notes

- Write one short sentence in `notes` for every 0. Name the claim, not a feeling ("says 60-day refund window; passage says 30").
- Two reviewers should score the same 20 answers before a large review. Discuss every disagreement on G and A until you agree on how to read the rubric, then split the rest.
- Do not change the rubric mid-review. Collect proposed changes and apply them to the next round so scores stay comparable.