Performance Degradation Metric for Contamination Evaluation
In test set contamination analysis for language models, performance degradation quantifies the disparity between performance on unseen items versus leaked items. It is calculated as the model's percentage score on non-contaminated questions minus its percentage score on contaminated questions: Evaluating this difference across benchmarks indicates whether exposure to evaluation questions during pre-training meaningfully skews reported capability metrics.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.2 Model Scaling and Capability Evaluation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Test Set Contamination Analysis - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Related
Measuring Test Set Contamination via Substring Matching
Performance Degradation Metric for Contamination Evaluation
Contamination as a Non-Substantive Confounder in GPT-4 Exam Evaluation
GPT-4 Performance on Academic and Professional Exams
Match each contamination analysis concept to its correct description.
Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using ___ matching.
Order the steps involved in performing a contamination analysis between benchmark questions and training data.
What is the primary rationale for stripping whitespace and symbols from both datasets before searching for matching text?
Performance Degradation Metric for Contamination Evaluation
Learn After
How is performance degradation calculated when evaluating test set contamination in language models?
In test set contamination analysis, non-contaminated questions represent leaked items, while contaminated questions represent unseen items.
In the context of test set contamination analysis, what does the performance degradation metric quantify?
A language model achieves a score of 62% on the non-contaminated questions of an evaluation benchmark and 88% on the contaminated questions of the same benchmark.
- Calculate the performance degradation metric for this evaluation.
- Interpret what this specific result indicates regarding the model's reported capability metrics.
Contamination as a Non-Substantive Confounder in GPT-4 Exam Evaluation