Benchmark Overfitting in NLP Evaluation
Models exploit benchmark quirks rather than develop real reasoning skills.
Tobias Ferrell
CRM Confessions — Page 3
Golden Datasets
Human Annotation Consistency in Eval Set Labeling
Unreliable human labels corrupt eval sets and mislead model rankings.
Stratified Sampling for Golden Dataset Construction
Stratified sampling reveals regressions in rare subgroups that average pass rates bury.
Priya Venkateswaran