Human Annotation Consistency in Eval Set Labeling

Human Annotation Consistency in Eval Set Labeling.

Annotation consistency as a data reliability problem

Inconsistent human labels do more than slow a project down. They corrupt the ground truth that every downstream model judgment, benchmark score, and release decision rests on. Most teams treat annotation as a scheduling problem, something to smooth over with better tooling or a tighter deadline, and that gets the priority backwards. The labels are the data. If the data is unreliable, every metric computed against it inherits that unreliability, no matter how sophisticated the model scoring against it happens to be.

Inter-annotator agreement, or IAA, tells a team whether its ground truth is real or manufactured. A high score means the guidelines are clear enough that independent people, working from the same instructions, land on the same answer. A low score means something upstream broke: the rubric doesn't cover the cases annotators are actually seeing, training didn't stick, or the task itself resists reliable labeling. None of those failure modes are cosmetic. They mean the "correct" label in the eval set is closer to a coin flip dressed up as consensus.

A specific and underappreciated failure lives here. A dataset labeled by generic reviewers, people with no grounding in the domain they're judging, can pass every accuracy check a team runs, because those checks are computed against the same flawed consensus that produced the errors. A dashboard never registers that kind of mistake. It becomes visible later, when a model meets a real-world edge case the eval set never actually tested, because the label it was scored against was wrong to begin with. Catching that requires a second signal: agreement between the people doing the labeling.

Where inter-annotator agreement metrics break down

Cohen's kappa is the standard starting point: chance-corrected agreement between two raters, adjusted for the agreement you'd expect from random guessing alone. A kappa above 0.80 counts as strong, 0.60 to 0.80 as substantial, and anything below 0.60 signals that the rubric needs work, not that the annotators are careless. Teams miss that distinction constantly, and treating a low kappa as a personnel problem instead of a rubric diagnosis fixes nothing. It just wastes a retraining cycle on people who were never the source of the noise.

Krippendorff's alpha extends kappa to more than two raters, which makes it the right choice for most production annotation pipelines rather than an academic nicety. Gwet's AC1 is a third option, and it behaves differently when label prevalence is extreme, so swapping it in for kappa without checking the underlying distribution can flatter a rubric that's actually broken, or condemn one that's fine.

The deeper trap is simpler and more common: treating raw percent agreement as though it were chance-corrected agreement. Raw percent agreement is not chance-corrected agreement, because it says nothing about how much of that overlap would have happened anyway, given how the labels are distributed, and reporting it alone overstates reliability, sometimes by a wide margin. Raw agreement says nothing about how much of that overlap would have happened anyway, given how the labels are distributed, and reporting it alone overstates reliability, sometimes by a wide margin. A team that publishes only that number is choosing the figure that looks best rather than the one that tells the truth about how trustworthy its ground truth is.

What empirical IAA scores look like in practice

Real annotation efforts put these thresholds in context, and the numbers are not flattering. A SemEval-2026 narrative similarity task, built specifically around deliberately difficult cases, recorded an individual Krippendorff's alpha of 0.33, which the authors describe as consistent with other subjective annotation work in the literature https://arxiv.org/pdf/2604.21782. That's a low score by any formal threshold, and the authors don't dress it up as anything else. Individual annotators, evaluated against latent oracle labels, hit roughly 78% accuracy on their own, but when two annotators happened to agree, accuracy on those same items jumped to roughly 92% https://arxiv.org/pdf/2604.21782. Agreement is a direct predictor of correctness here, and consensus requirements buy real reliability even on tasks where individual judgment alone runs shaky.

A 2026 study annotating 100 Reddit posts across 13 dimensions shows how much the choice of metric alone can shift the story a team tells about its own data. Cohen's kappa on those same annotations ranged from 0.37 to 0.60, squarely in the zone that signals rubric trouble. Report only the raw agreement number from that study, and the story sounds a lot rosier than the chance-corrected math supports.

Expertise doesn't rescue this either. A Nature study on clinical decision-making had 11 ICU consultants annotate the same patient data and found only fair agreement, a Fleiss' kappa of 0.383 https://www.getmaxim.ai/articles/guide-to-managing-human-annotation-in-ai-evaluation-best-practices/. These are trained specialists working inside their own domain, disagreeing at a rate that would fail most quality bars in a commercial annotation pipeline. Across NLP tasks generally, human-to-human kappa runs between 0.5 and 0.8, a band wide enough to span "ship it" and "start over on the rubric https://futureagi.com/blog/human-vs-llm-annotation-2025/."

Annotation errors, benchmark ranking propagation, and model evaluation corruption

Label errors in a test set aren't neutral background noise. They actively reshuffle which models look best, and the reshuffling can be dramatic at error rates most teams would consider tolerable. On ImageNet, correcting mislabeled test examples at mislabeling rates as low as 6% flipped the ranking between ResNet-18 and ResNet-50, with the smaller model coming out ahead https://www.emergentmind.com/topics/label-errors-in-test-sets. That is not a rounding error sitting quietly on a leaderboard.

The pattern holds outside image classification. The RePOPE revision of the POPE object hallucination benchmark corrected label errors and ambiguous annotations in the original test set, and F1 scores along with top model rankings shifted once those corrections landed. On MMLU, researchers found leading language models sometimes scored as well, or better, on questions whose original label was simply wrong as they did on verified-correct questions, a result that points toward contamination or memorization inflating scores in ways a clean benchmark would never surface. Across a review of 15 popular LLM benchmarks, sanitizing label errors changed the story GSM8K was telling entirely: noise in the original test set had buried performance differences between models, and once the labels were fixed, more capable models produced higher accuracy, as expected. The hierarchy the benchmark existed to reveal had been sitting there the whole time, hidden inside bad labels.

This risk does not spread evenly across a leaderboard. It concentrates hardest at the frontier, where the top few systems often sit within a point of each other. At that margin, the small percentage of gold labels that are simply wrong functions as the deciding vote. A benchmark built to rank the best models in the world can end up ranking the errors in its own annotation instead, and nobody notices until someone bothers to check the labels.

The rubric and workflow failures that create annotation inconsistency in the first place

A handful of concrete practices reliably produce this kind of noise, and none of them require anyone to act in bad faith. Using a single reviewer per item is the most basic one. Without overlap between annotators, there is no way to compute agreement at all, and no way to tell whether a strange model output reflects a strange model or a mislabeled example that produced it.

Writing the rubric after the data has already been collected does nearly as much damage. The guidelines end up retrofitted to whatever the first batch of annotators happened to do, and labels stop being comparable across model versions once that rubric inevitably gets revised again. Mixing training labels with evaluation labels quietly breaks the independence a benchmark needs: labels used to optimize a model should not also be the labels certifying the release that consumed them. Showing annotators which model produced which output biases the review toward whichever system the annotator already trusts, corrupting the evaluation before a single score gets computed.

Each failure survives fine in isolation. That is why teams let them slide. Stacked across a pipeline, though, these failures compound, and the resulting eval set can look clean on the surface while carrying a systematic bias that no accuracy check will ever catch, because the check is measuring against the same broken consensus.

Building annotation pipelines that produce reliable ground truth at scale

Most of this goes wrong at a specific point: treating annotation as a one-time labeling task instead of a closed-loop pipeline. A more durable structure traces each sample from raw data through the human decision, into an evaluator threshold, through a regression run, and finally to a release gate. Annotated rows don't get used once and discarded. They become golden datasets, regression evals, and holdout sets used specifically to check whether an LLM-as-judge prompt behaves the way it's supposed to. When reviewer agreement drops below the target threshold in a system built this way, the next action is a rubric review. That order of operations, checking the rubric before blaming the model, is what separates a pipeline that gets better over time from one that just accumulates noise quietly.

Qualification comes before any of that. Annotators shouldn't get near production data until they've cleared golden task tests, items with known correct answers, and only the ones scoring above a defined threshold move forward. Guidelines aren't a document written once and filed away. They stay alive, revised continuously against annotator feedback, and they need decision trees for edge cases, side-by-side examples of correct and incorrect labels, and explicit protocols for what to do when a case is genuinely ambiguous. Layer a quality control structure on top of that, where more experienced annotators review the newer ones and regular accuracy screens catch systematic errors early, and drift gets caught before it ever reaches a benchmark.

LLM-as-judge has moved from an interesting research direction to the default approach for evaluating free-form generation, and the shift is not hype. The G-Eval paper and Prometheus 2 both showed that a calibrated LLM judge, given a clear rubric, tracks human preference more closely than older metrics like BLEU or ROUGE on most tasks. By May 2026, frontier judge models were clearing a kappa of 0.7 or better on most tasks, and that number changes where human annotation time gets spent best: not labeling every single trace by hand, but calibrating the judge and triaging the cases where judge and human disagree https://futureagi.com/glossary/human-annotation/.

The aggregate numbers back this claim further than most teams expect. The best LLM annotator in one comparison reached 79.9% macro-level agreement, edging out human-to-human agreement at 79.2%, with a marginally higher Krippendorff's alpha, 0.606 against 0.585 for human pairs https://arxiv.org/html/2606.02255v1. Model-assisted annotation is not a fallback for when human labeling runs too slow or too expensive. On these numbers, it's slightly more stable against the gold standard than human annotators are against each other, and treating it as a lesser substitute misreads what the data shows.

That result comes with a real limit, and it shouldn't get smoothed over to make the previous paragraph sound cleaner. Judge-to-human Spearman correlation can clear 0.5 when the rubric is explicit and well specified, but it drops below 0.3 on subjective, open-ended writing tasks https://futureagi.com/glossary/human-annotation/. LLM consistency tracks the clarity of the rubric, not some general capacity for judgment. Where a task has a clean, defined answer, a calibrated judge can match or beat human consistency. Where a task is genuinely a matter of taste or interpretation, no rubric precise enough exists yet to replace a human reader, and pretending otherwise is how a team ends up shipping an eval set that looks rigorous and isn't.

The business cost of letting annotation inconsistency go unmanaged

AI spending is forecast to surpass $2 trillion in 2026, up 37% year over year, and when investment scales at that pace, the cost of poor data quality, annotation included, scales right alongside it https://www.ibm.com/think/insights/cost-of-poor-data-quality. Gartner predicts organizations will abandon 60% of AI projects through 2026 for lack of AI-ready data https://www.ewsolutions.com/cost-of-poor-data-quality/. A flawed eval set built on inconsistent labels is one of the concrete, traceable ways "AI-ready" fails in practice, not an abstract data-governance concern sitting off to the side.

Gartner's broader research puts the average annual cost of poor data quality at $12.9 million per organization, and none of these figures isolate annotation as its own line item https://www.integrate.io/blog/data-quality-improvement-stats-from-etl/ https://kili-technology.com/blog/2026-data-labeling-guide-for-enterprises-build-high-performing-ai-with-expert-data. The mechanism matters more than the accounting here. A model can only be as trustworthy as the eval set that certified it, and an eval set can only be as trustworthy as the agreement behind its labels. Skipping the IAA check just defers discovery of the risk sitting inside a benchmark to the moment a released model meets a case its "ground truth" never actually tested.

That deferred discovery carries a price tag in enterprise deployment. The gap between lab benchmark scores and real-world performance for enterprise agentic AI systems reaches 37%, and the cost of reaching similar accuracy across different deployments can vary by as much as 50x https://kili-technology.com/blog/ai-benchmarks-guide-the-top-evaluations-in-2026-and-why-theyre-not-enough. Of 54 reviewed works using manual annotation, 29% gave no information at all about inter-annotator agreement https://arxiv.org/pdf/2312.07228. Nearly a third of published annotation efforts can't tell a reader whether their ground truth means anything.

The rest of the figures point the same direction without needing to pile up. An IBM Institute for Business Value report found that 43% of chief operations officers name data quality as their top data priority, and over a quarter of organizations estimate annual losses above $5 million from poor data quality, with 7% reporting $25 million or more https://www.ibm.com/think/insights/cost-of-poor-data-quality. As much as 80% of an AI project's time goes to data preparation, gathering, organizing, and labeling, which means the annotation step is not a side task bolted onto model development https://www.herohunt.ai/blog/how-to-assess-human-data-labelers-screening-the-ai-workforce-guide/. It is most of the project. 43% of chief operations officers identify data quality issues as their most significant data priority, and over a quarter of organizations estimate losses exceeding $5 million annually from poor data quality, with 7% reporting $25 million or more. A 2026 study annotating 100 Reddit posts found a raw inter-annotator agreement range of 0.74–0.95 across 13 dimensions https://arxiv.org/pdf/2601.20747. A 2026 study annotating 100 Reddit posts found a Cohen's Kappa inter-annotator agreement range of 0.37–0.60 across 13 dimensions https://arxiv.org/pdf/2601.20747. A 2026 study annotating 100 Reddit posts found a Gwet's AC1 inter-annotator agreement range of 0.64–0.94 across 13 dimensions https://arxiv.org/pdf/2601.20747. The RLTHF approach with 6% human annotation achieved a 15.9× return on investment improvement compared to random human annotation https://kili-technology.com/blog/2026-data-labeling-guide-for-enterprises-build-high-performing-ai-with-expert-data. Poor data quality causes companies to lose 15–25% of annual revenue, according to MIT Sloan Management Review and Cork University Business School research https://www.integrate.io/blog/data-quality-improvement-stats-from-etl/. Just 6% of targeted human annotation delivered significant ROI https://kili-technology.com/blog/2026-data-labeling-guide-for-enterprises-build-high-performing-ai-with-expert-data.