Bounded Benchmark Validity: Two Question Families and 21/18 Unique Held-Out Targets Cap Statistical Power
Even after stricter leakage controls, the benchmark-validity scope of this paper's headline retrieval claims is explicitly bounded. The prerequisite datasets instantiate only two question families, and the target-concept-disjoint test splits expose only 21 unique held-out targets on LectureBank-Full and 18 on MOOC-CS. With this few unique targets, paired-bootstrap intervals over per-target stability are necessarily wide and the statistical power to distinguish adaptive from fixed-depth hierarchical retrieval at R@10 is limited. The evidence therefore applies to curated, template-based prerequisite QA rather than broad educational or open-domain graph retrieval.
0
1
Tags
Science
Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Related
LectureBank-Full Decomposition: Diffusion+Quotas Drive ~18 R@10 Points; Contrast Gating Adds At Most ~1 Point (Statistically Tied)
Bounded Benchmark Validity: Two Question Families and 21/18 Unique Held-Out Targets Cap Statistical Power
Restrained Claim Scope: No Automatic-Judge Validation, No Semantic-Evasion Claim, No Robust End-to-End QA Gains
Language-Matched Seeding as a Prerequisite for Graph-Expansion Gains
HotpotQA External-Validity Probe: Adaptive Depth Does Not Transfer to a Denser Non-Prerequisite Graph (FullWiki-1k: Flat 93.4 / Hier 92.9 / Adaptive 94.0 R@10)
Complementary Contribution Scope Over the Graph-RAG Evaluation-First Lineage
Bounded Benchmark Validity: Two Question Families and 21/18 Unique Held-Out Targets Cap Statistical Power