LLM-as-Judge Calibration and Bias Auditing

DoorDash Engineering has published its own approach to using large language models to evaluate search result pages, a documented case of a pattern that has moved well past the research stage and into daily production operations. That migration is the subject of this piece: LLM-as-judge systems, in which one model scores, compares, or ranks the output of another, have become the default quality-assurance layer across production AI pipelines, and they now operate at a scale and a cost structure that human review simply cannot match. The mechanism itself is straightforward. A judge model takes a rubric and a candidate output and returns a structured verdict, an integer, a float, a binary pass or fail, a pairwise preference, or a multi-dimension JSON object, and it does this across millions of inferences a day at a fraction of what human annotation would cost.

That economics is precisely what has pushed LLM judges into infrastructure. Frontier-tier judges already undercut human annotation by a wide margin, and distilled, flash-tier judges push that gap further still, which makes scoring at scale not just feasible but close to mandatory for any team running a pipeline of meaningful size. The consequence is that a judge running on millions of inferences a day, if it drifts out of calibration, does not announce itself through a scatter of obviously bad individual calls. A miscalibrated judge shifts the baseline quietly, and that shift becomes visible only when a user files a ticket or an incident review goes looking for a root cause.

Consider a pattern that recurs across teams running judge-scored pipelines in production: a system scores helpfulness at 0.91 for an entire quarter, dashboards stay green, and nothing looks wrong. Then a minor model version bump moves the judge's mean score and narrows the distribution around it. Continuous integration keeps passing, because the numbers still sit inside the expected band. Months later, an agent error surfaces downstream, and the retrospective finds the actual cause in the judge's change: the judge changed, the rubric never did, and the score stopped measuring what the team believed it was measuring. If a meaningful error rate holds across millions of daily evaluations, a correspondingly large volume of outputs gets misclassified: some get waved through validation, others get flagged in false alerts, and all of it happens at machine speed before a human ever looks at a single example.

Regulatory pressure is closing in on exactly this gap. The EU AI Act and comparable regimes now require demonstrable evaluation of AI system outputs, and a judge that disagrees with human raters in a consistent, directional way is an audit finding. The thing that makes this hard to catch on an ordinary engineering dashboard is that nothing about a miscalibrated judge looks broken from the outside: the scores stay inside their normal range, the pass rates look stable, and the system presents every outward sign of health right up until the failure it was supposed to catch finally gets reported by someone else. That invisibility is the reason a dedicated calibration discipline, rather than a one-time validation step, is necessary for any pipeline that depends on an LLM judge.

The consistency–bias paradox: why a judge that looks reliable can still be systematically wrong

A judge that returns the same score on the same input every time it's run is not, for that reason alone, a judge worth trusting. High test-retest reliability and genuine accuracy are independent properties, and a system can score at the top of one while failing the other. If a judge is perfectly consistent and perfectly wrong, it will pass every reproducibility check a team runs, because those checks measure agreement with itself, not agreement with the ground truth the rubric was meant to capture. The correct way to describe this is consistent bias: a judge that returns the same distorted answer every time is not catching a fluke, it is executing a stable error.

The scale of this problem has now been measured directly. A 2026 large-scale evaluation covering 21 judges from nine providers (Norman, Rivera, and Hughes) found that kappa deflation, the gap between raw exact-match agreement and the chance-corrected discrimination that Cohen's kappa actually measures, appears across every evaluated model. Raw agreement figures overstate how much real discriminative power a judge has, and judge rankings built on one benchmark shift substantially when the same judges are run against a different one. That instability in ranking means a judge validated as "best in class" on one evaluation set carries no guarantee of holding that rank anywhere else.

The aggregate numbers that the field has leaned on tell a reassuring story, but that story doesn't survive contact with harder tests. The original MT-Bench agreement figure from Zheng et al. (2023) holds up as an accurate description of average judge performance across a broad benchmark. But frontier models, run against the more targeted JudgeBiasBench (Hongli Zhou and colleagues), exceeded 50% error rates on advanced bias probes designed to isolate specific failure patterns. An aggregate agreement figure just tells you how a judge performs across a wide spread of easy and hard cases averaged together. It says nothing about how that same judge handles the borderline, ambiguous outputs that a production model actually generates in volume, which is the number that matters operationally and the number aggregate benchmarks are least equipped to reveal.

Work on FairJudge (Bo Yang and colleagues, February 2026) traces this breakdown to three failure modes that compound one another, so it isn't scattered or random but structural. The first is adaptivity failure: a judge configured for general chat quality gets pointed at code review, medical summarization, and creative writing alike, applying one rubric to tasks that have nothing in common, and ends up measuring something other than what each task actually requires. The second is non-semantic bias, in which position, length, formatting, and even which model family produced the output shape the verdict in ways that have nothing to do with the content's actual quality. The third is cross-mode inconsistency, or Score-Comparison Inconsistency: a response that earns a higher score in pointwise evaluation can still lose a head-to-head pairwise comparison against a response it outscored, producing circular preference chains where A beats B, B beats C, and C beats A. These three failures feed each other. A judge working from the wrong rubric is also the one most likely to pick up non-semantic cues as a substitute for real judgment, and that same judge then produces pointwise and pairwise scores that contradict each other, with each failure mode making the other two harder to isolate and fix.

The five named biases and their measured effect sizes

LLM-judge bias is five distinct, independently documented failure modes, each with its own mechanism, its own measured effect size, and its own detection procedure, and treating all five as one undifferentiated "bias problem" is precisely why generic mitigation strategies keep failing to hold.

Position bias is the best-studied of the five. Zheng et al. (2024, MT-Bench) found that in pairwise comparison, the response placed in slot A wins more often than chance alone would predict, and this swings the winrate on close calls by 10 to 15 points. The mechanism is structural rather than incidental: autoregressive scoring means the judge reads response A before it reads response B, and that early context shapes the verdict before the second response even enters the picture. Simply instructing the judge to "consider both responses equally" does almost nothing to fix it, with measured effects from that kind of prompt-level instruction running close to zero. Lin Shi and colleagues (IJCNLP 2025) studied position bias across a large set of judges and evaluation instances with a choice-pair framework that sorts outcomes into position-consistent, primacy-preferred, and recency-preferred behavioral signatures, and they found the bias varies meaningfully across both judges and tasks.

Verbosity bias is triggered by length and surface markers of effort. Wang et al. (2023) found that GPT-4, Claude, and PaLM-2 judges inflated their preference for longer outputs by 15 to 30 points, even when the researchers held actual answer quality constant across comparisons. The judge is pattern-matching on signals that correlate loosely with effort, response length, dense lists, explicit hedging, citation count, none of which function as a direct proxy for whether the content is actually correct or useful.

Self-preference, sometimes called family bias, shows a judge favoring outputs from models in its own lineage. Zheng et al. (2024) identified this self-enhancement effect as a systematic property of LLM judges in the MT-Bench study, but that paper itself doesn't reliably establish a specific numeric range for same-family score inflation. The underlying mechanism is plausible on its face: models from the same family share training data, alignment procedures, and stylistic habits, which gives a judge more surface-level reasons to prefer outputs that resemble its own. Research published at EMNLP 2025 under the title "Beyond the Surface: Measuring Self-Preference in LLM Judgments" draws a distinction that matters for anyone trying to act on this: some instances of a model favoring its own output reflect a real quality advantage the model has over others, so not every instance of self-favoring is bias. The failure that actually causes harm is narrower and more specific: a judge that declines to penalize errors made by models in its own family, even when those errors are visible and real.

Format bias responds to how content is presented. Testing by the RAND JRH found that simple formatting changes, markdown versus plain text, bulleted versus prose, disrupted the consistency of judges that had already cleared standard accuracy checks: a judge can look reliable under one formatting convention and fall apart under another without any change to the substance being evaluated.

Anchoring bias is the newest entry in the taxonomy, documented in 2026 by Kapetanovic, Altwlkany, Mercep, and Duricic at Infobip, presented at CIKM '26 in Rome. Their study tested three prompt conditions, no metadata, revision framing, and anchored metadata carrying revision count, attempt number, and prior score, across 185,271 successful evaluations, and found that seven of the eight evaluated models showed prior scores included merely as context metadata systematically pulling subsequent judgments toward those earlier values. The measured effect reached a Cohen's d of 0.71 for the anchored-metadata condition, and conventional standards in behavioral research call that a large effect. The mechanism traces directly to Tversky and Kahneman's anchoring-and-adjustment heuristic: the judge treats a visible prior score as a starting point and then fails to adjust far enough away from it, even when the content in front of it warrants a very different verdict. Neither chain-of-thought prompting nor an explicit warning telling the judge to disregard prior metadata eliminated the effect; the warning condition improved paired accuracy relative to baseline, but the bias persisted regardless. This matters most in iterative refinement loops and multi-agent pipelines, where a score from an earlier pass often sits in the context window of a later evaluation call that is assumed, incorrectly, to be independent of it.

Detecting each bias before it corrupts production scores

Detecting any of these five biases starts from the same prerequisite: a human-labeled gold set, because without one there is no independent reference point against which a judge's disagreement can even be measured. A workable gold set runs 200 to 500 hand-labeled traces per workload per rubric, with each trace labeled by two or three human reviewers and inter-annotator agreement tracked through Cohen's kappa. You need to refresh that set on a quarterly cadence, retiring its oldest entries in favor of fresh production traces, so it keeps tracking the actual distribution of work the judge sees rather than a snapshot of what the pipeline looked like months earlier.

To test for position bias, you run every pair through the judge twice, once in each order, and measure the percentage of pairs where the verdict flips purely because of which response came first, alongside the mean score shift on pairs that don't flip. When the flip rate climbs above a low threshold, you know order, not content, is driving a meaningful share of the judge's verdicts. Verbosity bias is isolated by sorting calibration pairs into length-matched buckets and length-mismatched buckets and scoring both: if the winrate or score gap between genuinely high-quality and low-quality answers narrows sharply once length is controlled for, length rather than quality is doing most of the work in the scores the team sees day to day. Self-preference is surfaced by cross-family scoring, running the same evaluation set through a judge from the same model family as the content's generator and separately through a judge from a different family, then comparing the resulting score distributions; the approach from the EMNLP 2025 self-preference research adds a gold judge from an entirely separate model family specifically to expose where the two verdicts diverge, since divergence concentrated on same-family outputs is itself the bias signal. Format bias is tested by re-scoring identical content rendered in an alternative format, markdown against plain text, bulleted lists against prose, and comparing the verdicts that come back; the RAND JRH methodology treats formatting changes, paraphrasing, and verbosity shifts as perturbation tests to run specifically against judges that have already passed standard accuracy checks, since passing those checks is what makes a hidden format sensitivity so easy to miss. Anchoring bias is tested with the Infobip CIKM '26 methodology directly: run the same fixed texts with known ground truth through three prompt conditions, no metadata at all, revision framing without a prior score, and anchored metadata that includes revision count, attempt number, and the prior score itself, and compare how far the verdicts move across those three conditions. A judge that holds steady across all three is behaving independently. A judge whose scores drift toward whatever prior value it was shown is exhibiting the exact failure Kapetanovic and colleagues measured at a Cohen's d of 0.71, and it needs to be treated as a production risk.