Canonical Prerequisite Splits Are Heavily Templated: 92/80 LectureBank-Full and 68/60 MOOC-CS Train-Test Overlaps
The canonical prerequisite splits used to train and evaluate prerequisite-QA systems are heavily templated, producing measurable train-test leakage. On LectureBank-Full, the canonical splits share exact train-test questions and shared train-test target concepts. On MOOC-CS, the canonical splits share exact train-test questions and shared train-test target concepts. These overlap counts are what motivate the paper's question-disjoint and target-concept-disjoint controls: without them, headline retrieval numbers on the canonical splits would partially reflect train-test template reuse rather than generalization.
0
1
Tags
Science
Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Related
LectureBank-Full Target-Disjoint R@10 Result (n=164): Diffusion Gain Survives, Adaptive Tied
MOOC-CS Target-Disjoint R@10 Result (n=114): Negative Case, Adaptive Tied
Adaptive Depth Gating Stays Tied to Hierarchical Baseline Under Target-Disjoint Control
Canonical Prerequisite Splits Are Heavily Templated: 92/80 LectureBank-Full and 68/60 MOOC-CS Train-Test Overlaps
LectureBank-Full Configuration Used in Hierarchical Prerequisite RAG (208 Concepts, 899 Edges, 1,421 QA)
MOOC-CS Configuration Used in Hierarchical Prerequisite RAG (225 Concepts, 516 Edges, 1,016 QA)
Canonical Prerequisite Splits Are Heavily Templated: 92/80 LectureBank-Full and 68/60 MOOC-CS Train-Test Overlaps
QASC Rebuilt as Directed Science-Fact Graph (16,444 Nodes, 25,590 Edges) Used Only for Validation Retrieval
Induced HotpotQA Slice as Auxiliary Stress Test in Supplement