Benchmark Validity for Multilingual and Code-Switching Models
Current benchmarks mismeasure multilingual AI ability.
Casimir Wojcik
Section
Golden Datasets
8 stories in Golden Datasets.
Synthetic Data Generation for Low-Resource Eval Scenarios
Synthetic test data helps teams catch AI failures before production reaches real users.
Casimir Wojcik
Eval Set Coverage Metrics Beyond Accuracy
Accuracy masks which failure modes your eval set actually covers.
Priya Venkateswaran
Version Control and Lineage Tracking for Eval Datasets
Eval datasets drift silently from production; version control and lineage tracking catch it.
Priya Venkateswaran
Domain Shift Detection in Training vs Evaluation Distributions
Models drift silently in production when training and real-world data diverge.
Casimir Wojcik
Human Annotation Consistency in Eval Set Labeling
Unreliable human labels corrupt eval sets and mislead model rankings.
Casimir Wojcik
Benchmark Overfitting in NLP Evaluation
Models exploit benchmark quirks rather than develop real reasoning skills.
Tobias Ferrell
Stratified Sampling for Golden Dataset Construction
Stratified sampling reveals regressions in rare subgroups that average pass rates bury.
Priya Venkateswaran