In this paper, **confidence intervals** and **paired system comparisons** on **LectureBank-Full**, **MOOC-CS**, and **QASC** are computed with **$$5{,}000$$ paired bootstrap resamples over questions**: on each resample, both systems being compared are re-evaluated on exactly the same resampled question indices, and the empirical distribution of per-resample paired deltas yields the 95% percentile CIs reported in the headline tables. To control the family-wise error rate when many system pairs are compared at once, **Holm–Bonferroni corrections are applied within comparison families** (the set of system pairs reported together in one comparison block), and the corrected results are presented in the **supplementary tables** rather than in the main headline numbers.

Statistical Protocol for Hierarchical Prerequisite Graph RAG: 5,000 Paired Bootstrap Resamples and Holm–Bonferroni

The paper's **primary retrieval outcomes** are **R@$$10$$** on **LectureBank-Full** and **MOOC-CS**, and **validation R@$$10$$** on **QASC** (whose official test split withholds evidence). **R@$$5$$** and **P@$$10$$** are reported as secondary metrics. Fixing R@$$10$$ as the headline cutoff matches the strict-parity contract's invariant $$k = 10$$; adding R@$$5$$ provides a stricter recall cutoff and P@$$10$$ provides a precision-style view of the same top-$$k$$ window.

Primary and Secondary Retrieval Metrics in Hierarchical Prerequisite Graph RAG

In *Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls*, **latency** is measured under a fixed protocol designed to remove caching and warm-start confounds. The protocol requires **Apple M4 Max CPU-only hardware**, **disabled caches**, a **warm-up phase**, **200 measured queries**, and **three repeats** of the full measurement. The reported $$\pm$$ values represent the **mean within-repeat per-query standard deviation** taken from the cited microbench JSONs. Thus, the dispersion captures run-to-run variation within a repeat rather than across-repeat variance. Reporting latency under this robust CPU-only protocol makes the timings directly comparable across the evaluated retrieval systems and datasets.

CPU-Only Latency Protocol in Auditable Strict-Parity Evaluation

This organizer groups the paper content from Experimental Setup > Metrics and Statistics.

Google

This organizer groups the paper's Experimental Setup material.

Experimental Setup in Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

This organizer groups the paper content from Experimental Setup > Datasets: the two curated prerequisite benchmarks (LectureBank-Full and MOOC-CS), the rebuilt QASC directed science-fact graph used only for validation retrieval, the templated-split overlap counts that motivate leakage controls, and the bounded held-out target counts under the strictest leakage control.

Datasets (Experimental Setup) in Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

This organizer groups the paper's baseline systems from Experimental Setup > Baselines. The strict-parity baselines all share the same encoder, the same dense-seed pool size $$m$$ and retrieval cutoff $$k$$, the same data split, and the same exact-ID matching rule, so any comparison isolates the choice of retrieval/graph policy. A multilingual encoder plus CJK-only query rewrite is included only as a control experiment on MOOC-CS and is not treated as a headline strict-parity comparator.

Learn Before

Related

Learn After