Benchmark Validity for Multilingual and Code-Switching Models

A performance gap and a measurement gap are not the same problem, and multilingual evaluation has been treating the second as though it were the first. A performance gap means a model falls short in some languages and better training, more data, or better architecture can close it over time. If you have a measurement gap, the scores used to judge that progress cannot be trusted, no matter how good the underlying model gets. The data suggest the field is sitting on a measurement gap it has mistaken for a performance gap. STEM-related task scores correlate strongly with human judgment, in the range of 0.70 to 0.85, but traditional NLP tasks like question answering, the kind most often cited as proof of multilingual competence, show correlations as low as 0.11 and rarely above 0.30. If you want to claim multilingual ability, the tasks you reach for are the ones whose scores least match what a human evaluator would actually conclude. An analysis of seven multilingual European benchmarks looked at how European-language models get judged today, found four major structural challenges, and concluded that benchmarks need to be culturally aware and rigorously validated for their scores to be trustworthy. So this conclusion does not describe a field with room for incremental improvement. It describes a field whose central measurement tool misleads more than it informs, in exactly the places where the stakes of getting it wrong are highest.

High-Resource Bias in Benchmark Construction

The skew toward a handful of languages does not happen after benchmarks are built. It happens in the decision of who builds them and what they build them from. An analysis covering more than 2,000 multilingual benchmarks found that development is concentrated in five countries, China, India, Germany, the UK, and the USA, together accounting for at least 40% of multilingual benchmark development¹²³⁴⁵. Even in many high-resource languages that dominate benchmark counts, high-value domains like healthcare and law remain underrepresented. This is a story about where research funding, academic infrastructure, and publication incentives already concentrate, reproduced one benchmark at a time. The tooling built to run these evaluations carries the same imbalance forward. The software layer meant to ease evaluation ends up reinforcing the same gaps already in the data, closing the loop: the benchmarks used to decide which languages need more attention are themselves built almost entirely on the languages that already have it. The clearest evidence of how deep that gap runs is the existence of EKA-EVAL, a unified evaluation framework for low-resource multilingual LLMs that its authors describe, to the best of their knowledge, as the first suite to offer comprehensive coverage of low-resource languages in a single platform. A claim like that, made in good faith by researchers building new infrastructure, is itself a measurement of how long this gap has gone unaddressed. If comprehensive low-resource coverage is still novel enough to be a paper's headline contribution, you know the field has spent years building benchmarks that only work for the languages that needed them least.

Why machine-translated test data corrupts the evaluation signal

Machine translation looks like the obvious shortcut for extending an English benchmark to dozens of other languages at a fraction of the cost of building something new. The shortcut breaks the thing it is supposed to produce. The European multilingual benchmarks analysis identifies two separate mechanisms by which this happens, and both change what the test is measuring rather than simply adding noise to it. The first is mechanical: in multiple-choice formats, an incorrect or incomplete translation does not just make one option wrong, it alters the entire set of choices a model is reasoning over, so a single translation error changes what capability the question is actually testing. If a model answers a mistranslated multiple-choice item, it succeeds or fails at a different task than the one the benchmark claims to measure, created accidentally by the translation. The second mechanism is cultural, and automated translation cannot fix it no matter how accurate the wording comes out. A widely used benchmark for testing broad knowledge contains questions spanning high-school to professional difficulty tied specifically to one country's context: its history, its legal system, its religious traditions as taught in its schools. Translate those questions word for word into another language and the words change, but the assumption that a test-taker shares a frame of reference with a student from that original country does not. Without a human adapting the content rather than just the language, the question becomes meaningless to the culture it now claims to test. The 2,000+ benchmark study puts a number on the downstream cost: translated benchmarks align far less well with local human judgments than benchmarks built from localized content in the first place, which undercuts the basic premise that translating a test makes its scores comparable across languages. Both also cost far more than running a script, so the cheap automated approach remains the default, and the problem recurs across benchmark after benchmark.

How data contamination turns benchmark scores into memorization artifacts

Contamination is the third structural threat, and it behaves differently in multilingual settings than it does in English-only evaluation, where the problem has already received substantial attention. When a benchmark's test items leak into a model's training data, a high score no longer measures reasoning; it measures recall. Translation makes that leakage harder to catch rather than easier. A single English test item, once translated into a dozen languages and scattered across web pages, forums, and documentation in the course of ordinary internet use, becomes a dozen surface forms of the same contaminated item, any one of which can end up in a training corpus without matching the original closely enough for standard overlap detection to flag it. A paraphrased or translated version of a benchmark question evades the kind of checks built to catch verbatim copying, while still inflating the score of any model that happened to see it during training. Benchmark saturation compounds the problem. A benchmark score only predicts how a model will perform in production when the test set is free of training contamination and when the benchmark has not saturated to the point where score differences between models are too small to mean anything. Saturation and contamination are identified as the two primary reasons benchmark scores fail to predict how a model will actually behave once deployed. A systematic review of contamination studies confirmed that when models train on web-scale corpora, the risk rises that their training data already contains the benchmark questions used to evaluate them, inflating reported performance across the board. In multilingual evaluation specifically, this means a procurement team comparing models on translated benchmarks may be rewarding the model that memorized the most translated test items, not the model that generalizes best to the actual languages it will serve in deployment.

What code-switching evaluation reveals about the deeper design failure

Code-switching, the practice of moving between two or more languages within a single sentence or conversation, is the setting where the structural weaknesses already described become unmistakable, because the metrics in use were built for a different problem. Word error rate and character error rate, the standard measures imported directly from monolingual speech recognition, have no cross-language standardization and measure transcription accuracy in isolation, with no way to capture what that transcription error costs downstream, in a booking system, a customer service transcript, or a voice assistant. Worse, aggregate WER systematically under-weights the embedded-language tokens, the minority language mixed into an otherwise dominant-language utterance, and that is usually the exact language a speaker is relying on the system to get right. A model can score well on aggregate WER while consistently failing the specific words that carry the most meaning for a bilingual speaker, and the benchmark will never show it. Massively multilingual pre-training systems are evaluated on the assumption that an utterance belongs to one language at a time, so the evaluation design has no mechanism for detecting the exact failure it should exist to catch. A related finding from research on multilingual multi-agent planning makes the same point in a different domain: planning-grounding failures, where task-critical information gets lost as a user's request is converted into an executable plan, make up a growing share of failed task executions as language-resource availability drops, with the pattern visible even in medium-resource languages and most severe in low-resource ones. Standard benchmarks built on monolingual assumptions cannot see this failure mode because they were never designed to look for it. The infrastructure gap shows in which languages get covered. AfriSwitch, introduced in 2026, was built in direct response to the limits of LinCE and GLUE-CoS, two earlier centralized efforts that proved shared code-switching benchmarks were valuable but remained text-only and included no African language pairs, leaving an entire continent's worth of real-world code-switching outside the benchmark infrastructure. CoSE-E, also from 2026, is an enterprise code-switching ASR benchmark that spans five language pairs, and it proposes a multidimensional evaluation framework built for enterprise use cases. That CoSE-E counts as new is itself informative: enterprise-grade evaluation for code-switched speech simply did not exist before it, despite the fact that enterprise voice systems have been handling mixed-language speech for years.

How benchmark invalidity distorts model selection and deployment decisions

Every structural flaw described so far collapses into one practical outcome: organizations choosing models for global products are making those choices on scores that do not reflect how the models will behave once real users in real locales start using them. MGSM, a benchmark that tests mathematical reasoning using 250 grade-school math problems translated into Bengali, Chinese, French, German, Japanese, Russian, Spanish, Swahili, Telugu, and Thai, illustrates the failure cleanly. A model can pass MGSM with a strong aggregate score and still fail badly on a specific locale split once deployed, because the benchmark's design has no way to isolate and surface that locale-specific weakness before it reaches a user. The benchmark passes the model. The model fails the user. The benchmark's own design is why that gap stays invisible until production. The multilingual multi-agent planning research offers a direct counterexample of what a more diagnostic approach can reveal. TART, Taxonomy-guided Actionable Task Representation, is a structured representation protocol that makes the taxonomy of planning failures explicit to the planner and downstream sub-agents, and it raises a state-of-the-art system's accuracy by 5.6 percentage points averaged across multiple languages on multilingual GAIA. A standard benchmark would have missed that improvement, because it only reports whether a task succeeded without diagnosing where in the pipeline it broke down. PolyWorkBench, introduced in 2026 to evaluate multilingual long-horizon LLM agents, exists for the same reason: most task-completion benchmarks operate only in English, leaving the exact deployment scenario that matters for agent-based multilingual products uncovered until PolyWorkBench filled the space. The LLM-as-judge paradigm, now widely used to automate multilingual evaluation at scale, is letting a further distortion into the pipeline. The judge models doing the scoring carry the same high-resource language bias as the models they judge, so the measurement layer inherits the bias it is supposed to catch and correct. If a team relies on LLM-as-judge scores to rank candidate models for a multilingual product, it is getting a second opinion from an evaluator with the same blind spots as the system under review.

Where the field is heading on multilingual benchmark validity

A structurally valid multilingual benchmark needs localized content built for the culture it tests rather than translated from English, a design resistant to contamination, coverage that includes low-resource languages as a priority rather than an afterthought, and metrics calibrated to mixed-language reality rather than borrowed from monolingual evaluation. No benchmark available today meets all four criteria at once. The 2,000+ benchmark study lays out six guiding principles for effective multilingual benchmarking and names five research directions still open, including better low-resource language coverage, genuine localization rather than translation, and contamination-resistant design that can be built efficiently rather than only at great expense. Even the fixes already in motion carry real costs. Dynamic benchmarks, refreshed continuously so that old test items cannot leak into new training runs, prevent contamination directly, but a fast refresh cycle paired with AI-assisted translation can compound cultural bias faster than human reviewers can catch and correct it, building up a kind of verification debt that nobody is tracking yet. Human-in-the-loop verification and iterative translation ranking work, and the research confirms it, but the same resource cost that makes them effective is the reason the cheaper automated shortcut became the industry default. TART points toward a more useful general principle than any single benchmark does: evaluation that shows where and why a system failed is worth more to a practitioner than a leaderboard number that only shows whether it failed. For a team selecting a model for a global product, the right question is not which benchmark produced the highest score. That benchmark's design assumptions, its language-resource coverage, its domain, its handling of code-switching, and whether its test data was localized or merely translated, must match the actual conditions the product will run in. A benchmark built on those assumptions is worth trusting. One built on the assumptions this piece has laid out is worth checking before it decides what gets deployed.