Learn Before
Activity (Process)

Paired Bootstrap Significance Testing and Confidence Intervals for Retrieval Evaluation (Smucker, Allan, and Carterette, 2007)

The paired bootstrap for retrieval evaluation is a resampling procedure that operates on per-query score pairs. For two retrieval systems AA and BB evaluated on the same set of nn queries, one forms the vector of per-query differences di=sA(qi)sB(qi)d_i = s_A(q_i) - s_B(q_i) in a retrieval metric (e.g., MAP, Recall@kk). A bootstrap replicate is generated by sampling nn queries with replacement and recomputing the mean of did_i on the resampled set; repeating this BB times (typically thousands) yields a bootstrap distribution of the per-query mean difference. A two-sided pp-value can be read from this distribution, and a paired bootstrap confidence interval for the true mean difference is obtained by taking the appropriate empirical quantiles (e.g., the 2.5% and 97.5% quantiles for a 95% CI). Smucker, Allan, and Carterette (2007) compared this paired bootstrap test to the paired tt-test, Wilcoxon signed-rank, sign test, and Fisher's randomization test on TREC ad-hoc runs and found the paired bootstrap, paired tt-test, and randomization test to be in close agreement, while the sign and Wilcoxon tests were less reliable. The paired bootstrap is therefore one of the recommended IR significance procedures and is the canonical source for paired bootstrap CIs over per-query retrieval scores.

The width of the resulting CI scales roughly with σ/n\sigma / \sqrt{n}, where σ\sigma is the standard deviation of the per-query differences and nn is the number of queries (or, in target-disjoint IR settings, the number of held-out targets). Narrow query/target pools therefore produce wide intervals, which directly bounds the smallest per-query effect size the procedure can resolve at a given confidence level.

0

1

Updated 2026-05-18

Contributors are:

Who are from:

Tags

Science

Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

Related