Stratified Sampling for Golden Dataset Construction

A golden dataset is a stand-in for reality: a fixed, labeled set of examples that a model gets scored against, over and over, so a team can tell whether a change made things better or worse. A stand-in only works if it stands for the right thing, and most golden sets fail at exactly that job. They get built by one engineer, in one afternoon, out of the ten cases that happen to come to mind. That produces a sample of one person's imagination. A 2026 engineering guide warns that a golden set built from one engineer's intuition ends up testing a system that does not exist, and the CI gate reads green because nothing in the set challenges the actual failure modes.

Part of the problem is treating three different artifacts as one thing. A testing golden dataset catches what breaks. A fine-tuning golden dataset teaches a model what good output looks like. An evaluation golden dataset measures what matters to the business or the user. These jobs pull in different directions, and no amount of extra data fixes a set that was built for one job and pressed into service for another. Aggregate pass rates make this worse by hiding localized regressions: a model can lose ten points on refund disputes while gaining five on FAQ traffic, and the blended number looks flat. Without a stable reference set, that kind of comparison gets confounded by traffic volume, seasonal shifts in what users ask, and changes in user behavior that have nothing to do with the model itself.

What stratified sampling does that random sampling cannot

Stratified sampling is a statistical technique that splits a population into subgroups, or strata, and then samples from each subgroup in proportion to its share of the whole. Applied to a golden dataset, it means the training split, the validation split, and the test split each contain roughly the same mix of intents, personas, and edge cases as the full pool of examples. Random sampling makes no such promise. Pulling cases at random from a support log can leave a rare but important class severely underrepresented or absent entirely, purely by chance.

That's the core failure: random sampling can produce splits that misrepresent the population it was drawn from, especially when a dataset is small or a class is rare. Two different random draws from the same pool can produce splits with meaningfully different class distributions, making pass-rate comparisons unreliable. Stratification removes that noise by guaranteeing every partition is a faithful miniature of the whole, common cases and rare cases both, in proportion.

Overrepresenting the easy majority class inflates precision, recall, and F1, while underrepresenting a minority group distorts those same metrics in the other direction. Overrepresent the easy majority class and the success rate looks inflated. Underrepresent a minority group, say, non-English speakers or a small but high-value enterprise segment, and a real regression in that group goes completely undetected, since it barely moves the blended score. A benchmark that can't see a regression measures the majority class and calls it quality. It's measuring the majority class and calling it quality.

The four-bucket model: how to partition a golden dataset so every gap becomes visible

The 2026 engineering guide referenced above lays out a four-bucket structure for a serious golden set, and calls anything short of it "a vibe check dressed up as a test suite." Dropping a bucket leaves the CI gate showing green, but an entire category of failure goes untested. The 2026 engineering guide's four-bucket structure breaks down into four categories, and three of them each prove something the others cannot.

Bucket one, the stratified production sample, makes up 60% of the set. The guide describes pulling roughly 1,500 raw traces and downsampling them into a 300-case target for this bucket. The stratification runs along three axes at once: intent (refund, status check, complaint, escalation, FAQ), persona (new user, power user, enterprise admin, someone in a regulated vertical), and retrieval shape (full context, partial context, missing context, contradictory context). In a RAG system, skipping the retrieval-shape axis means missing the single most common cause of production failure, which is bad context reaching a model that then reasons correctly on bad information. In a RAG system, skipping the retrieval-shape axis means missing the single most common cause of production failure, which is bad context reaching a model that then reasons correctly on bad information. The fix is a coverage matrix: every cell with meaningful production volume gets at least 5 to 10 cases, and any cell with zero volume belongs in bucket three instead, not here. What this bucket proves is narrow but load-bearing: the system handles the traffic it actually gets.

Bucket two, the adversarial library, is 15% of the set. Production traces underrepresent attacks that haven't happened yet, so this bucket has to be built from other sources: public adversarial corpora like HarmBench and AdvBench, security frameworks such as the OWASP LLM Top 10, a scanner harvest run against production traces, and synthetic adversarial generation. Every case in this bucket needs a tag for source and sub-category (jailbreak, injection, malformed input, poisoned retrieval), because blending adversarial scores into the same number as production scores muddies both. This bucket proves the system holds under attack, which is a different claim than proving it handles normal traffic.

Bucket three, deliberate edge cases, is another 15%. These are hand-written by domain experts to probe the long tail a model hasn't seen yet: inputs that probe capability boundaries a model hasn't encountered in normal production traffic. This bucket has nothing to do with current traffic frequency; it's a deliberate stress test of capability boundaries, built on the assumption that today's rare case is tomorrow's normal one. What it proves is that the system handles the long tail it hasn't encountered yet, which no amount of production sampling can substitute for, since production sampling by definition only shows what has already happened.

Sizing the dataset: the 100–500 example sweet spot

Size follows a rough three-tier pattern. A set as small as 20 examples is too small to catch anything but the most obvious breakage, and false confidence follows cheaply built sets of that scale. Somewhere between 100 and 500 examples is the sweet spot for a mid-sized production system, provided the examples are stratified by domain and difficulty rather than just piled up. Past a few hundred examples, the practical move is often to split into stratified sub-samples or separate golden sets per segment rather than maintaining one enormous monolith, since a single oversized set tends to make updates harder to sustain.

There's a statistical reason the 100–500 range works, and it comes from margin-of-error math rather than convention. Statistical sizing guidance from AI evaluation practitioners puts the sample size needed at around 246 if a team expects an 80% pass rate on some slice and wants a 5% margin of error at 95% confidence. That number moves depending on the expected pass rate and how tight the margin needs to be, but it explains why "a few hundred" keeps appearing as the answer across different domains: it's close to the point where the standard error curve flattens out enough that adding more examples stops buying much additional confidence.

The range assumes 1 to 2 annotators doing the labeling. Crowdsourced labeling changes the math, since disagreement rates in crowdsourced labels run 40% to 50%, which forces majority voting or expert review and adds roughly 3 to 4 times the cost of the simpler setup. A NIST finding, cited in industry writing on this topic, adds a useful correction to the instinct that bigger is always better: a smaller, carefully composed test set can match the coverage of a much larger one, because what determines coverage is class representation and scenario diversity, not raw count. A 500-example set that hits every critical input class will catch more regressions than a much larger set skewed toward the easy majority.

Label quality and inter-annotator agreement: the human judgment layer that makes the benchmark credible

None of the sizing or stratification work matters if the labels themselves are wrong, and labels are wrong more often than most teams assume. Research on major ML benchmarks found roughly 3.3% label errors. That's not a rounding error, label mistakes at that rate are enough to flip published rankings between models. The ground truth a benchmark rests on is not automatically trustworthy just because it's been published or widely used.

Inter-annotator agreement is the operational check against this. Measured with Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha depending on how many annotators and label types are involved, agreement below 0.40 counts as poor, the range from 0.40 up to 0.75 is intermediate to good, and 0.75 or above is excellent. AI evaluation practitioners set the practical action line at 0.7: fall below that, and the move is to revise the annotation guidelines before labeling anything else, not to push forward and hope the noise averages out. Per dotdatalabs.ai, if two trained humans can't agree on a label, an automated scorer trained or prompted to mimic that judgment won't be reliable either. Fixing the guideline has to come before scaling the labeling effort, not after.

Consensus annotation is the standard fix: two or three annotators label the same example independently, and disagreements get reconciled rather than broken by a single tiebreaker vote. The resulting label carries more weight than any one expert's opinion, and the disagreements themselves are useful signal, not noise to be discarded. A case where trained annotators split down the middle is revealing genuine ambiguity in the task definition, and that ambiguity needs resolving before the case earns a place in the golden set, not after.

Golden dataset staleness and the maintenance workflow that prevents silent score inflation

A golden dataset built once and left alone does not stay accurate. It goes stale in at least six distinct ways, and each one needs a different repair, since treating them as one generic "the data is old" problem misses what's actually broken.

Traffic drift happens when the languages, intents, input lengths, customer segments, or abuse patterns showing up in production shift away from what the golden set captured. System drift happens on the other side: the prompts, model family, context window, retriever, or orchestration logic changed, so the golden set is now testing a system that no longer exists in that form. Knowledge drift is specific to retrieval-augmented systems, where reference documents expire or time-sensitive facts go out of date. Policy drift covers the cases where what counts as the correct refusal, escalation, tone, or business action has changed, often because of a business decision that had nothing to do with engineering. The engineering guide cited earlier describes label drift as quieter: reviewers start applying the same rubric differently over time, and a later annotator reading earlier guidance will interpret an edge case differently than the earlier annotator did. Saturation and contamination round out the list: saturation is what happens when engineers repeatedly tune against the same visible cases until nearly everything passes, and contamination is when holdout examples leak into prompts, training data, or few-shot examples, making the eval a memory test instead of a generalization test.

The maintenance model that answers all six is a three-pool structure rather than one flat list. Anchor regressions hold the severe historical failures and hard product invariants, stay stable unless the underlying contract changes, and support comparisons across time. Rolling representative cases approximate current production traffic and get periodically replaced through stratified sampling, with a floor set for rare but consequential strata so they don't get sampled away to zero, and deduplicated by conversation, semantic cluster, document, or account so a single incident burst doesn't dominate the set. Challenge cases probe new capabilities and adversarial inputs without pretending to represent traffic frequency, existing specifically to surface future failure rather than describe the present.

One trap deserves its own warning: when policy drift happens, the fix is never to quietly lower a passing threshold so the numbers look stable. The right sequence is to update the requirement, the rubric, the reference answer, and the affected cases first, and only then let the score reflect the new definition of correct. Skipping that step causes historical comparisons to mix two different definitions of correctness under one number, a worse failure than staleness itself, since it looks like consistency while actually being drift in disguise. Versioning closes the loop: every evaluation run needs a tag for the exact dataset version it ran against, and adding, removing, or relabeling a case should go through something like a pull request, not an ad-hoc edit that nobody can trace later.

Using production monitoring to feed the golden dataset continuously rather than rebuilding it periodically

Most teams' golden datasets and their production systems drift apart silently. New failure modes appear in real traffic, nobody captures them into the golden set, and the benchmark keeps reporting numbers against a version of reality that no longer exists. AI observability tooling closes that gap by feeding new cases into the golden set as production monitoring detects novel failures, turning the dataset from a one-time artifact into something closer to a living product with its own maintenance cycle.

The adversarial bucket benefits from this in a specific, almost mechanical way. Running adversarial scanners against the raw bucket-one pull of production traces means any trace that trips a scanner is, by definition, a real attack that actually reached the system. Promote that trace into bucket two with the scanner's verdict attached as its label. That means the adversarial bucket grows automatically as production grows, without a separate curation sprint every quarter to go find new attack examples by hand.

The refresh cycle itself should run monthly, or tied to release cadence rather than the calendar, whichever comes first. Freeze a production sampling window, generate privacy-safe candidates drawn from successes, failures, and the uncertain cases sitting in between, cluster and deduplicate before anyone labels anything, and compare the resulting coverage against the current dataset before deciding what to add. The review batch that comes out of that comparison should lean toward new and underrepresented slices, since those are exactly the places where a stale golden set would have kept reporting a green light while the real system quietly drifted somewhere else.