ModelRefs / Pairwise Comparison — Prompt Pattern
Pairwise Comparison — Prompt Pattern
Have the judge compare two outputs head-to-head instead of scoring individually. Replace absolute scores with A vs B preference, then aggregate to Elo.
Overview
Replace absolute scores with A vs B preference, then aggregate to Elo. Reduces scale-clustering bias.
When to use it: Absolute LLM-as-judge scores cluster and don't discriminate.
Pattern details
| Pattern class | evaluation |
|---|---|
| Difficulty | intermediate |
| Also known as | preference judging, elo eval |
| Last reviewed | 2026-06-07 |
Known failure modes
- Position bias — Favors A or B. Mitigation: Swap order; average.
When not to use it
- Pairwise eval with N > 4 candidates per comparison.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pairwise Comparison — Prompt Pattern.
Frequently asked questions
When should I use the Pairwise Comparison prompt pattern?
Absolute LLM-as-judge scores cluster and don't discriminate.
What are common failure modes of Pairwise Comparison?
Position bias
Does Pairwise Comparison work across models?
Yes when the anatomy is preserved — see model compatibility matrix on the pattern page.