LLM-as-Judge and Its Failure Modes
Position bias, self-preference, verbosity bias and miscalibration. How to build a judge that correlates with human labels, and how to know when it has stopped.
Position bias, self-preference, verbosity bias and miscalibration. How to build a judge that correlates with human labels, and how to know when it has stopped.
The rubric score moved from 6.8 to 7.9 the week you rewrote the support assistant's prompt, and nobody argued with it, because a model graded all two thousand answers and a model has no stake in whose work is whose. Three weeks later the support lead pastes a thread into your channel. The new answers are longer, warmer, better formatted, and wrong about the refund window.
The judge did not lie to you. It measured something real — it measured length and polish, because that is what your rubric rewarded, and nobody ever checked. By the end of this lesson you will know the four ways a judge quietly goes wrong, the specific test that catches each one, and the single number you print next to every judge-graded score so that nobody — including you — mistakes it for a measurement.
LLM-as-judge is the practice of making a model call whose output you record as a measurement. You send it one or two generated answers and a rubric, and you store what comes back as if it were a test result.
It comes in two shapes and they fail differently. A pointwise judge sees one answer and returns an absolute score against a rubric — 7 out of 10, or "grounded / partly grounded / ungrounded". A pairwise judge sees two answers to the same input and says which is better. Pairwise is more reliable per decision, because comparing is an easier task than scoring, but it gives you no absolute level: you learn that the new prompt beats the old one, never whether either is good. Pointwise gives you a level you can track across releases and is far easier to miscalibrate without noticing.
Either way, treat the judge as an instrument. A thermometer that reads three degrees high is a perfectly good thermometer once you know it reads three degrees high. One that has never been checked against boiling water is a decoration you hang next to your data. Every failure mode below is a characterisation you have not done yet.
Position bias is the judge's verdict depending on which slot an answer occupies rather than on the answer. Put the same two responses in front of the same judge twice, swapping which one is labelled A, and a biased judge picks the same slot both times — which means it contradicts itself about which text is better.
There is no mechanism forcing a model to treat two candidates symmetrically. They sit at different distances from the instruction, from the question, and from the end of the prompt, and the comparison is next-token prediction over that whole asymmetric arrangement. Symmetry has to be measured, never assumed.
The measurement is the swap test: run every comparison in both orders and count how often the judge agrees with itself.
Read the two numbers together.
A judge you can use.
It agrees with itself almost every time, and it has no detectable preference for either end of the prompt.
A coin with an opinion about slots.
Thirty-eight percent of your pairwise results were decided by the argument order of your loop, and the second number tells you which end it leans towards.
Bad — one order per pair, so the loop's argument order silently becomes part of the result.
Good — both orders, and a disagreement between them is reported as what it is.
With a judge that favours the second slot 60/40, the first version hands roughly ten points of win rate to whichever argument you happened to pass second — and you always pass the new prompt second, because that is how the diff reads. The version you ship is chosen by a habit of typing.
Self-preference bias is a judge scoring text from its own model family higher than equivalent text from another. It is not loyalty — the judge has no idea who wrote either answer, and telling it to ignore authorship changes nothing, because authorship is not what it is responding to. It is responding to style: phrasing, hedging, structure, the shape of a good answer as that family renders it. Text a model would have produced itself reads as correct to it.
You catch it with a cross-judge matrix. Take the same set of answers written by three model families, score them all against one fixed baseline, and change only who is judging.
| Judge | Family A wins | Family B wins | Family C wins |
|---|---|---|---|
| A | 61% | 48% | 44% |
| B | 47% | 58% | 46% |
| C | 45% | 49% | 59% |
Every judge ranks its own family first by roughly twelve points, and the three of them disagree about who is second. The off- diagonal cells are the honest ones. If the diagonal is flat, you have found a genuinely neutral judge for this task; keep it.
Verbosity bias is a longer answer scoring higher for being longer. Formatting bias is the same effect from headings, bullets and bold text. Both are easy to believe you have avoided and easy to prove you have not, because there is a clean experiment: change the presentation and nothing else.
Take answers your judge has already scored, expand them with words that carry no new information, and re-score them.
A shift of 0.6 on a ten-point scale, from text that says nothing new, is your judge quoting you a price per word. Run the same experiment for formatting alone — identical sentences, wrapped in two headings and a bullet list — and you usually find a smaller version of the same number.
The reason this one is urgent rather than interesting: as soon as you start tuning prompts against the judge, you are optimising against its biases too, and a length preference is the easiest gradient in the room. Every iteration makes your answers longer and your score higher. The product gets worse while the graph goes up.
Bad — an unanchored scale with an instruction that rewards bulk directly.
Good — one axis, a written description per level, and length ruled out by name.
The first rubric has no shared meaning for the gap between 6 and 7, so the judge fills it with whatever correlates with effort — and "be thorough" tells it that more is better about the answer as much as about the review. You end up measuring word count with extra steps.
Miscalibration here is not being wrong on average; it is having no spread. You run five hundred answers through a ten-point judge and 92% of them come back as 7 or 8. The eval still produces a number every night, and that number cannot distinguish your best release from your worst, because the instrument has no resolution in the range where your system actually lives.
The test for resolution is a negative control: a small set of deliberately broken answers that any competent reviewer would reject instantly.
Plant ten to twenty of these and run them on every judge change. They are the smoke detector, and they cost four model calls.
When the spread is gone, shrinking the scale usually restores it. Ten unlabelled points invite a judge to hover near the middle; three levels with a written description each force a decision. Better still, switch that comparison to pairwise, where the resolution comes from the comparison rather than from the judge's private sense of what an 8 means.
Here is the discipline that turns all of the above from anxiety into a process. The judge is a model, and every model in your stack has an accuracy you are expected to know. For a judge, that accuracy is its agreement with human labels, and until you have measured it you do not have an evaluation — you have a second model's opinion, stored in a database.
Label a set by hand first. Two to five hundred items, drawn from real traffic, stratified so the hard and rare cases are present rather than swamped by easy ones. Label them before anyone looks at judge output, because a reviewer who sees the machine's answer first will agree with it more often than they should.
Raw agreement flatters you whenever the classes are unbalanced: a judge that passes everything scores 90% raw agreement on traffic that is 90% fine. Cohen's kappa corrects for the agreement you would get by chance, and linear weights tell it that confusing 1 with 3 is worse than confusing 1 with 2 on an ordinal scale.
Two things make kappa readable. The first is the ceiling: have two people label the same hundred items and compute the kappa between them. If your humans agree at 0.74, a judge at 0.71 is close to as good as the task allows; if humans agree at 0.95, the same 0.71 is a bad judge. The second is the confusion matrix, because judge errors are never symmetric. Split them: a human FAIL that the judge called PASS is the error that ships a regression, and a human PASS the judge called FAIL only wastes an engineer's afternoon. Track the false-pass rate separately and set your threshold on that.
From then on, the agreement travels with the score. Not "the
candidate passed 74%", but "the candidate passed 74%, judge
groundedness-v4, kappa 0.71 against 400 labels, measured on
2026-05-02". A score without that line is unfalsifiable.
And the moment the judge changes, the line expires. A new judge model version, an edited rubric, a different scale, even a shift in the population of inputs — each one invalidates the agreement you measured and, with it, every historical score you were comparing against.
Bad — the judge is a floating alias and a file on disk, so it changes without a version change.
Good — pinned, versioned, and carrying the agreement it was last measured at.
With the first version, you soften the rubric on Tuesday to stop the judge complaining about markdown, the score jumps four points on Wednesday, and you attribute it to the product change that shipped in between. The regression that was really there is now invisible, and the trend line covering it goes back months.
A judge is the most expensive and least trustworthy way to check anything that can be checked another way, so the decision comes before the rubric.
If a deterministic check exists, use it and do not hold a vote.
Does the JSON parse; does it validate against the schema; does
the generated SQL run; is the cited document ID actually in the
retrieved set; does the test suite pass; is the extracted total
equal to the total on the invoice. A judge scoring "is this valid
JSON" is a slower, costlier, noisier json.loads that sometimes
says yes to broken input.
A judge is worth its cost when three things hold at once: the quality is genuinely open-ended, so no exact check exists; the volume is high enough that people cannot look at all of it; and a single wrong grade is cheap, because it is one row in an aggregate rather than a decision about a customer. Groundedness, tone, "does this answer the question that was asked", "is the explanation appropriate for a beginner" — those are judge work.
People stay in the loop where the opposite holds: low volume, high stakes, or novelty. The first two hundred outputs of a capability you have never evaluated before, anything with legal or medical or financial consequence, and every item where the judge disagrees with itself across a swap. Those disagreements are also your best source of new labels.
The layering that works in practice: exact checks as a gate, so malformed output never reaches a model call; the judge on what survives the gate; and humans on a stratified sample plus every swap disagreement, feeding the label set that keeps the judge honest. Budget for it — a judge over two thousand items on every merge is two thousand model calls per run, and it is routinely the most expensive job in the suite. A cheaper, weaker judge model is a real option, but choose it by re-measuring agreement, not by price alone: the cheap judge that drops kappa from 0.71 to 0.44 has not saved you anything, it has stopped measuring.
The failure that survives everything above is the one where the judge and the system under test are wrong in the same direction.
Your assistant answers a question about a command-line tool and invents a flag that does not exist — plausible name, plausible syntax, exactly the flag the tool ought to have. The judge, built on the same model family and trained on the same corner of the internet, finds it plausible too. It marks the answer correct. The item scores 9. The swap test is clean, the length experiment is flat, the score is high and stable, and the answer is a fabrication.
This is worse than noise, and the reason is uncomfortable. Noise is symmetric and announces itself as disagreement. A shared blind spot announces itself as agreement and a good score. Your evaluation gets quieter as it gets wronger.
Three counters, in order of how much they buy you. Use a judge from a different model family than the system under test, which does not eliminate a shared blind spot but stops the two of them being identical. Make sure your human-labelled set contains items from the domain where you suspect the overlap — a blind spot the labels never cover is a blind spot kappa cannot see. And structurally, the strongest move: stop asking the judge what it knows and start asking it what it can see.
"Is this correct?" asks the judge for knowledge it may share with the thing it is grading. "Is every claim in this text supported by that document?" asks it to compare two strings in front of it, which is a far easier task and one where its own beliefs stop being the deciding input. Wherever you can supply ground truth in the prompt, the judge becomes a verifier, and a verifier has far less room to be confidently wrong alongside you.
You can now treat a judge the way you treat any other dependency: pin it, version it, measure its accuracy against a labelled set, publish that accuracy beside every score it produces, and re-measure the moment anything about it moves. Swap-test your pairwise comparisons, null-edit for length and formatting, keep negative controls in the suite, and never let a model family adjudicate its own bake-off.
The obvious hole is where those human labels come from. Human-in-the-Loop Design answers it: how to build review that people can sustain, how confidence thresholds decide what reaches them, and what happens to your label quality when a reviewer on their four-hundredth item starts agreeing with whatever the screen already says. A judge is only as good as the labels it was calibrated against, and those labels are a system with failure modes of their own.
Start with the swap test, today, on the judge you already have in CI. It is twenty lines and one extra call per comparison, and the consistency number it prints tells you immediately whether the last three months of pairwise results were measurements or coin flips.
Rate this support reply from 1 to 10 on overall quality.
Be thorough and comprehensive in your assessment.Score this support reply on one axis: does it answer the
customer's question using only facts present in the ticket?
3 = answers fully, and every fact appears in the ticket
2 = answers, but introduces a detail the ticket does not
contain
1 = does not answer the question, or contradicts the ticket
Length, tone and formatting are not criteria. First quote
the sentence that decided the score, then give the number.Below is a documentation excerpt and an assistant's answer.
List every factual claim the answer makes. For each claim,
mark SUPPORTED and quote the line of the excerpt that
supports it, or mark UNSUPPORTED. A claim that is merely
consistent with the excerpt, or that you believe from your
own knowledge, is UNSUPPORTED.
Then: SUPPORTED_ALL if every claim is supported, otherwise
UNSUPPORTED_ANY.SHOULD THIS BE A JUDGE AT ALL
exact check exists? use it - json.loads beats a rubric
low volume or high stakes? use people; judge only as a filter
open-ended + high volume? judge, then measure it below
CALIBRATION - run all of these before the first release
swap test every pair in both orders
-> consistency >= 0.90, slot win rate ~ 0.50
length null-edit same claims, twice the words
-> mean score shift near zero
format null-edit same sentences, headings and bullets added
-> mean score shift near zero
cross-family re-judge with a different model family
-> the ranking of variants must not change
negative control 10-20 deliberately broken answers
-> every one must land at the bottom
score histogram >80% of items on two adjacent levels means
no resolution - shorten the scale or go
pairwise
AGREEMENT - the number you print next to every eval score
human labels 200-500 real items, labelled before the
judge runs and before anyone sees its output
raw agreement % identical - inflated by class imbalance
cohen kappa chance-corrected; linear weights if ordinal
human ceiling two labellers on the same 100 items; judge
kappa only means something against that
false-pass rate human FAIL + judge PASS - the error that
ships a regression; threshold on this one
confusion matrix read it - judge errors are never symmetric
RE-MEASURE - each of these invalidates every historical score
judge model version changed
judge prompt or rubric edited
scale or label set changed
input population shifted
REPORT LIKE THIS
candidate 74% pass | judge groundedness-v4 | kappa 0.71
vs 400 human labels | measured 2026-05-02def swap_test(pairs, judge):
"""Every pair both ways -> (consistency, first-slot rate)."""
consistent = 0
first_slot_wins = 0
for baseline, candidate in pairs:
forward = judge(option_a=baseline, option_b=candidate)
reverse = judge(option_a=candidate, option_b=baseline)
first_slot_wins += (forward == "A") + (reverse == "A")
# candidate won iff forward said B, or reverse said A
if (forward == "B") == (reverse == "A"):
consistent += 1
total = len(pairs)
return consistent / total, first_slot_wins / (2 * total)def compare(baseline, candidate):
return judge(option_a=baseline, option_b=candidate)def compare(baseline, candidate):
forward = judge(option_a=baseline, option_b=candidate)
reverse = judge(option_a=candidate, option_b=baseline)
if (forward == "B") != (reverse == "A"):
return "tie" # the ordering decided it, not the text
return "candidate" if forward == "B" else "baseline"def inflate(answer: str) -> str:
"""Same claims, twice the words, zero new information."""
return (
"Thanks for reaching out about this.\n\n"
"## What is happening\n\n"
f"{answer}\n\n"
"## Why this matters\n\n"
"Knowing how this works helps you resolve the issue "
"quickly and avoid running into it again later.\n"
)
original = [judge_score(a) for a in answers]
padded = [judge_score(inflate(a)) for a in answers]
print(mean(padded) - mean(original)) # 0.0 is what you wantNEGATIVE_CONTROLS = [
truncate_mid_sentence(golden["refund_window"]),
answer_for(golden["shipping_delay"]), # right shape, wrong
# question entirely
contradict_source(golden["refund_window"]),
strip_to_greeting(golden["password_reset"]),
]
# Every one of these must land at the bottom of the scale.
# If they score 6, the judge cannot see failure, and the 8s
# on your real answers mean nothing.from sklearn.metrics import cohen_kappa_score, confusion_matrix
raw = sum(h == j for h, j in zip(human, judge)) / len(human)
kappa = cohen_kappa_score(human, judge, weights="linear")
print(f"raw agreement {raw:.2f} kappa {kappa:.2f}")
print(confusion_matrix(human, judge, labels=[1, 2, 3]))JUDGE = JudgeConfig(
prompt=open("judge_prompt.txt").read(),
model=settings.judge_model_latest,
)JUDGE = JudgeConfig(
prompt_version="groundedness-v4",
model=settings.judge_model_pinned, # exact version, never
# a floating alias
kappa_vs_humans=0.71, # 400 labels
measured_on="2026-05-02",
)