**Confidence intervals and paired comparisons** in this paper use **$$B = 5{,}000$$ paired bootstrap resamples** drawn **over questions** (each replicate resamples the same set of questions for both compared systems). Resampling questions — rather than per-system scores — preserves the **paired structure** that makes the test sensitive to per-question performance differences between systems.

Google

The **paired bootstrap** is a hypothesis-testing and confidence-interval procedure for comparing two procedures (or systems) $$A$$ and $$B$$ evaluated on the **same** $$n$$ items. Let $$d_i = m(A,i) - m(B,i)$$ be the per-item paired difference of a performance metric $$m$$, with observed mean $$\bar d$$. For $$b = 1,\dots,B$$ replicates (with $$B$$ typically in the thousands), draw a bootstrap index set $$I^{*(b)} \subset \{1,\dots,n\}$$ of size $$n$$ **with replacement** and recompute the paired statistic on the **same** resampled items for both systems, e.g. $$\bar d^{*(b)} = \tfrac{1}{n}\sum_{i\in I^{*(b)}} d_i.$$ The empirical distribution $$\{\bar d^{*(b)}\}_{b=1}^{B}$$ approximates the sampling distribution of the paired difference, from which a **percentile confidence interval** $$[\bar d^{*(\alpha/2)},\bar d^{*(1-\alpha/2)}]$$ is read off and a **two-sided p-value** can be derived (e.g., as the bootstrap probability that the difference reverses sign). Resampling **item indices** rather than the two systems independently preserves the pairing structure, which removes between-item variance and gives more power than an unpaired test.

Paired Bootstrap Resampling for Hypothesis Testing

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Learn Before

Related