Learn Before
Measuring Test Set Contamination via Substring Matching
Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using substring matching. Prior to searching for matching substrings between the evaluation questions and the training corpus, both datasets are preprocessed by stripping out all whitespace and symbols to prevent trivial formatting variations from masking overlapping text.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.2 Model Scaling and Capability Evaluation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Test Set Contamination Analysis - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Ch.1 Foundation Model Capabilities and Benchmarking - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Benchmark Contamination Analysis - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Related
Measuring Test Set Contamination via Substring Matching
Performance Degradation Metric for Contamination Evaluation
Contamination as a Non-Substantive Confounder in GPT-4 Exam Evaluation
GPT-4 Performance on Academic and Professional Exams
Contamination as a Non-Substantive Confounder in GPT-4 Exam Evaluation
Performance Degradation Metric for Contamination Evaluation
Measuring Test Set Contamination via Substring Matching
Learn After
Match each contamination analysis concept to its correct description.
Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using ___ matching.
Order the steps involved in performing a contamination analysis between benchmark questions and training data.
What is the primary rationale for stripping whitespace and symbols from both datasets before searching for matching text?
Performance Degradation Metric for Contamination Evaluation