Annotation guidelines for pairwise comparison of LLM responses
Labeler guidelines for comparing two model responses side by side: a priority order for deciding (safety, correctness, instructions, helpfulness, clarity), label definitions, per-response flags, rationale examples and edge cases.
File in Data by AISkillHub Library
What it does
Pairwise comparison (is A or B better?) is the most common way teams compare two prompt or model versions with human judgment, and the most common way it goes wrong is inconsistent labelers. One person rewards length, another rewards confidence, a third calls everything a tie. These guidelines give every annotator the same decision procedure.
**Core of the guidelines**
- **Decide what a good answer needs before reading the responses.** - **Compare in strict priority order and stop at the first clear difference**: safety and policy, then correctness, then instruction following, then helpfulness, then clarity. One significant factual error outweighs any amount of polish. - **Length is not a criterion.** A longer response wins only when the extra content is needed. - **Labels**: `A`, `B`, `tie`, `both_bad`, with `slight` or `clear` strength and definitions for each. Tie is not a way to avoid a hard call. - **Flags per response**: `factual_error`, `instruction_miss`, `refusal_unneeded`, `unsafe`, `incomplete`, `hallucinated_source`, `tone`, `verbose`. - **Rationale**: one or two sentences naming the deciding difference, with good and bad examples ("B states the limit is 100/min; reference says 60" versus "A is better").
**Edge cases covered**
Both responses refuse; code you can and cannot run; different but valid approaches; wrong-language replies; ambiguous prompts; recognizing which model wrote a response.
**Quality process**
Known-answer items for consistency checks, double-labeled items with disagreement review, a rule to stop and ask rather than guess, and three "do not" rules (no pasting items into other AI tools, no copying personal data, no changing labels after seeing others').
**What it will not do**
It is not a labeling tool and does not define your sampling or statistics. Pair it with whatever interface you use (a spreadsheet works) and randomize which version appears as A.
How to use it
1. Copy `docs/llm-output-annotation-guidelines.md` into your project or share it with annotators. 2. Adjust the flags and edge cases for your domain (for example, add a `formatting` flag if output format is critical). 3. Set up your labeling sheet or tool with the fields `preference`, `strength`, `flags_a`, `flags_b` and `rationale`. Randomize which version is shown as A for each item. 4. Run a calibration round: all annotators label the same 20 to 30 items, then discuss disagreements and update the guidelines where they were unclear. 5. Include known-answer and double-labeled items in every batch, and review disagreements regularly. 6. Report win rate with ties and `both_bad` shown separately; `both_bad` items are often the most useful for fixing prompts.
Files
Save each file at the path shown, relative to your project root.