University of Michigan - Ann Arbor

Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using substring matching. Prior to searching for matching substrings between the evaluation questions and the training corpus, both datasets are preprocessed by stripping out all whitespace and symbols to prevent trivial formatting variations from masking overlapping text.

Measuring Test Set Contamination via Substring Matching

Match each contamination analysis concept to its correct description.

Cross-contamination between evaluation benchmarks and large language model pre-training data can be measured using ___ matching.

Order the steps involved in performing a contamination analysis between benchmark questions and training data.

What is the primary rationale for stripping whitespace and symbols from both datasets before searching for matching text?

In test set contamination analysis for language models, performance degradation quantifies the disparity between performance on unseen items versus leaked items. It is calculated as the model's percentage score on non-contaminated questions minus its percentage score on contaminated questions:
$$\text{Degradation} = \text{Score}_{\text{non-contaminated}} - \text{Score}_{\text{contaminated}}$$
Evaluating this difference across benchmarks indicates whether exposure to evaluation questions during pre-training meaningfully skews reported capability metrics.

Learn Before

Related