Version Control and Lineage Tracking for Eval Datasets

Faithfulness holds at 0.87, task completion at 0.91, and the continuous integration gate stays green for six weeks straight. Then a customer-success thread surfaces a hallucination on a query tied to a March feature release, a case the golden set has zero examples resembling. Nothing in the evaluation pipeline caught this because nothing was built to catch it: the CI gate stayed green, the scores held steady, and no alarm fired while the eval quietly drifted away from what production traffic actually looked like. The model did not fail here. The eval infrastructure failed, because the dataset scoring it had stopped representing the world it was supposed to measure.

Three distinct ways an eval dataset drifts from production reality

Eval drift is not a single failure mode with one fix. Version control has to address at least three independent ways a dataset can decouple from reality, because any one of them can silently corrupt scores on its own. The first is input distribution drift: production queries change as new features ship and users adopt new vocabulary and traffic patterns shift, while the golden set sits frozen at whatever slice of the world it captured months earlier. The second is prompt-template drift: the wording of a prompt changes between eval runs without anyone logging the change, so a score delta that actually belongs to the prompt gets attributed to the model instead.

The third is retrieval-corpus drift, and it is the one with the clearest documented fix. In retrieval-augmented generation systems, the knowledge base version and the eval dataset version can fall out of sync, and The Neural Base's dataset lineage guide treats this as a specific, nameable failure: a knowledge base version mismatch invalidates both faithfulness and context precision scores. The fix that guide recommends is a hard stop: if the current knowledge base version does not match the lineage-recorded knowledge base version, the system should raise an error before the eval even runs. It must address at least three independent drift modes, since each one can silently corrupt scores on its own.

What "treating data with the same rigor as code" requires

The principle is easy to state: every dataset version gets an identifier, every change gets tracked, and any historical state can be reproduced on demand. Sysart's 2026 guide states this directly: treat data with the same rigor as code. The implementation is where most teams stall, because the tooling built for text diffs in a code repository was never designed for large binary datasets, and the infrastructure has to be built on purpose.

Versioning answers what data existed at a given point in time. Lineage answers how that data got used: which preprocessing ran on it, which training run consumed it, which eval run scored against it. Sysart's 2026 guide defines a complete lineage graph as four connected layers, raw data ingestion, transformation, training, and inference, and all four have to be captured or the chain stops being auditable at whichever layer got skipped.

For eval workflows specifically, The Neural Base's lineage guide gets concrete about what this looks like at production scale: compute a hash once, store it in a .lineage.json sidecar file next to the dataset, and load and compare that hash before every eval run. In distributed eval setups, every worker needs to validate it is reading the same upstream_commit or dataset_hash before it starts, because if half the workers in a multi-worker eval are scoring against a stale dataset version, the number means nothing even when each worker's math is correct.

If there is no metadata catalog sitting alongside the version store, none of this works. That catalog has to index every dataset version along with its lineage, its statistical profile (row counts, column distributions, null rates), and its quality scores, and it has to stay fast to query even when the underlying data sits on slow storage. And the lineage metadata itself needs version control: tracking changes to lineage graphs through Git-based repositories or the versioning built into modern data catalogs lets a team roll back a bad change, review how data moved through the system historically, and keep an audit trail that survives personnel turnover.

The tooling stack that implements eval dataset versioning in practice

No single product closes this loop.

DVC, Data Version Control, is open-source, and you can layer it directly on top of Git. It stores lightweight metafiles in the code repository while the actual data sits on separate storage, NFS, S3-compatible object storage, or HDFS, which keeps the Git repository small while preserving full version history even for datasets running hundreds of gigabytes. Sysart's 2026 guide identifies DVC as the most widely adopted open-source option for this job.

MLflow covers a different part of the stack: experiment tracking, the model registry, and lineage tracking, and it pairs naturally with DVC to connect data versions directly to the training runs that consumed them. LakeFS takes a different approach again, bringing a Git-like branching model directly to object storage systems like MinIO, Ceph, or other S3-compatible backends, so a team can branch, commit a data snapshot, and merge changes without ever putting the production eval set at risk. If your eval data lives in structured tables rather than files, Delta Lake and Apache Iceberg give you table-format versioning with time-travel queries, so you can reach any historical snapshot at the storage layer itself.

You can integrate OpenLineage and Apache Atlas into data processing frameworks to get automatic lineage capture, but they do not cover the same ground equally. Atlas only captures automatically for Hadoop-native sources such as Hive, Spark, and Sqoop, so anything outside that ecosystem needs custom engineering to get the same visibility. OpenLineage-compatible connectors for Apache Spark and dbt emit lineage events without manual instrumentation, but a custom Python pipeline still needs explicit instrumentation added by hand, typically a few lines of code per pipeline stage.

Storage strategy underlies all of this. A naive full-copy approach to versioning does not hold up once datasets reach production scale, because every version doubles the storage bill. Content-addressable storage, used by both DVC and LakeFS, stores only the unique data blocks that changed, so an incremental update to a dataset consumes storage proportional to what actually changed rather than the dataset's full size. Tiered storage policies then move older versions to cheaper storage on their own, so you keep history available without keeping it expensive. Label Your Data's 2025 checklist adds a fourth category to this stack alongside Git-integrated tools, lake-based systems, and versioned databases: workflow and observability tooling, treated not as an optional layer on top but as part of the versioning stack itself.

Why reproducibility and auditability are the same requirement

Reproducing an eval result and auditing a model decision require the same underlying capability: tracing a specific output back to the exact data and transformations that produced it.

Sysart's 2026 guide is direct about the stakes in regulated domains. For decisions in credit scoring, medical diagnostics, or insurance underwriting, auditors need to trace a specific prediction back to the training data and preprocessing logic that produced it. Without lineage tracking, that reconstruction can take weeks of manual archaeology through old logs and institutional memory. With it, the same reconstruction becomes a query that returns in seconds.

GDPR's right to erasure turns this from a theoretical concern into an operational one. Whether a model needs retraining after a subject exercises the right to erasure and has their data removed depends on an organization's ability to track which models trained on which data. The lineage graph is the only artifact that answers the question.

The NovaFabric paper frames the deeper difficulty here: an agent run is a graph of independently versioned, independently drifting parts, model weights, prompts, tools, the retrieved corpus, the runtime environment, and proving what the system actually did requires capturing the state of all of those parts at the exact moment the run happened. That requirement compounds the need for proof that the eval dataset actually matches the model it was run against: if the eval infrastructure cannot produce tamper-evident, replayable evidence of which dataset version ran against which model version, the scores it produces cannot satisfy an audit no matter how clean the underlying data is.

How poor lineage propagates into downstream systems

Once eval scores cannot be trusted, every system downstream of the model inherits that unreliability, and it inherits it at the speed and scale of automation rather than at the pace a human reviewer would catch it. The failure chain runs in a straight line: an untracked eval dataset drifts, offline scores stay green, a model with degraded production quality ships anyway, AI-driven workflows such as lead scoring, forecasting, content generation, and outreach sequencing act on that model's outputs, and the errors compound at machine speed before a human gets the chance to notice.

AI agents need more than raw access to data to act well. They need context: where the data came from, what transformations it passed through, what it actually means, and whether it can be trusted. Without lineage, an agent has no grounding for any of that, and it acts on what it's given regardless of whether what it's given is accurate.

The Bedrock case illustrates what closing that gap looks like in practice. The team had relied on manually mapped lineage to understand how data moved through its systems, and that process was slow and error-prone by nature. With OvalEdge's auto-lineage capabilities, the team could trace data across its environment and investigate quality issues much closer to where they originated, shortening the distance between where a problem started and where it surfaced.

This is the pattern behind a lot of stalled AI rollouts. Organizations that struggle to move AI projects from prototype to production often have a DataOps problem before they have an AI model problem: the models are ready to go, but the data infrastructure that supports them, including the eval infrastructure, is not. The same logic appears in CRM and RevOps systems running alongside AI deployments. When agents act on stale or incomplete records, they amplify errors at machine speed, producing misdirected outreach, inaccurate lead scores, and forecasts that miss before anyone catches the mistake. The eval dataset failure and the CRM data reliability failure are the same structural failure playing out in adjacent systems.

From alerting to autonomous correction: why detection alone is not enough

Lineage tracking that surfaces drift without resolving it leaves a team in the same reactive posture that created the problem. Detection is necessary, but it does not replace the work of correcting what it finds.

Sysart's 2026 guide points toward where this is heading: organizations running continuous data quality checks detect issues in real time and trigger corrective action automatically, rather than waiting for a periodic audit to catch what already went wrong. The same architecture applies directly to eval datasets. Version control and lineage tracking form the sensing layer. Autonomous agents that act on the signals that layer produces, flagging version mismatches, halting eval runs against stale datasets, proposing or applying corrections, form the response layer.

The Neural Base's lineage guide turns this into a hard engineering rule. When the validation function detects a knowledge base version mismatch, a warning is not enough. It should raise an exception and stop execution outright. The same principle holds for systems of record outside the eval pipeline. Agentic platforms built for CRMs and RevOps tools that continuously monitor records, detect duplicates and inconsistencies, and propose or apply evidence-based fixes on their own close the loop that alerting-only tools leave open.

This requirement does not change depending on whether the system of record is an eval dataset or a customer database. Clean, versioned, lineage-tracked data is the condition that has to be met before anything built on top of it, including an eval score that looks clean and a model that looks ready, can be trusted at all.