Measuring Test Set Contamination via Substring Matching
Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using substring matching. Prior to searching for matching substrings between the evaluation questions and the training corpus, both datasets are preprocessed by stripping out all whitespace and symbols to prevent trivial formatting variations from masking overlapping text.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.2 Model Scaling and Capability Evaluation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Test Set Contamination Analysis - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Learn After
Match each contamination analysis concept to its correct description.
Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using ___ matching.
Order the steps involved in performing a contamination analysis between benchmark questions and training data.
What is the primary rationale for stripping whitespace and symbols from both datasets before searching for matching text?
Performance Degradation Metric for Contamination Evaluation