On **LectureBank-Full**, the **paired delta** between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** in R@$$10$$ is only **$$+0.7$$ points**, with a 95% paired-bootstrap CI of **$$[-2.1, +3.6]$$**. Because the CI straddles zero, the two systems are statistically **tied** under strict parity, which is why the paper treats **depth gating as optional rather than essential** on this benchmark.

Google

**LectureBank-Full** is the expanded configuration of the LectureBank corpus used as a prerequisite-QA / retrieval benchmark. It contains $$1{,}717$$ English NLP-related lecture slide files and manual concept-pair prerequisite annotations over $$322$$ topics. The task is to retrieve, for a given target concept (query), the set of gold prerequisite concepts from a fixed candidate pool. Systems are scored with ranking metrics such as Recall@$$k$$ (R@$$k$$); R@$$10$$ measures the fraction of gold prerequisites recovered in the top-$$10$$ retrieved candidates. Standard splits hold out concept pairs for evaluation, and headline comparisons fix the encoder, candidate pool, cutoff, matching rule, and split policy.

LectureBank-Full Prerequisite QA Benchmark

This organizer keeps part 4 of the method details for Strict-Parity Evaluation Contract for Graph-Aware small enough for learners to study without a long flat child list.

Method Part 4: Strict-Parity Evaluation Contract for Graph-Aware (Auditable Strict-Parity Graph-RAG Paper)

This organizer keeps part 1 of the method details for Adaptive Depth Gating for Prerequisite Retrieval small enough for learners to study without a long flat child list.

Method Part 1: Adaptive Depth Gating for Prerequisite Retrieval (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups metrics, datasets, statistical protocol, and result evidence used by LectureBank-Full: Diffusion Plus Quotas Account For The Main Gain.

Evaluation Evidence for LectureBank-Full: Diffusion Plus Quotas Account For The Main Gain (Auditable Strict-Parity Graph-RAG Paper)

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

On the **LectureBank-Full** prerequisite-QA benchmark under the paper's strict-parity contract, **flat dense** retrieval reaches **R@$$10 = 55.5$$** (95% CI $$[51.9, 59.3]$$), a **fixed-depth hierarchical baseline** reaches **$$86.0$$** $$[81.9, 89.7]$$, and **Adaptive (heuristic) depth gating** reaches **$$86.7$$** $$[82.9, 90.6]$$. Because the encoder, candidate pool, cutoff $$k = 10$$, matching rule, and split are held constant, the $$\approx 30$$-point jump from $$55.5$$ to $$86.0$$ is attributable to **deterministic bidirectional diffusion with role-aware quotas**, not to adaptive depth control.

LectureBank-Full R@10 Gain from Diffusion and Role-Aware Quotas

In this paper, **LectureBank-Full** is consumed as a prerequisite graph with **$$208$$ concepts**, **$$899$$ prerequisite edges**, and **$$1{,}421$$ QA pairs** derived from instructional materials. Graph descriptors computed from the released graph and concept texts are: mean depth $$3.2$$, fraction of nodes with depth $$\geq 1$$ equal to $$90.4\%$$, mean degree $$8.6$$, and mean concept-text length $$66.3$$ tokens. The dataset is one of the two curated prerequisite benchmarks used for headline retrieval comparisons.

LectureBank-Full Configuration Used in Hierarchical Prerequisite RAG (208 Concepts, 899 Edges, 1,421 QA)

Under the **target-concept-disjoint** control on **LectureBank-Full** ($$n = 164$$), R@$$10$$ is **$$56.9\%$$** $$[53.9, 60.1]$$ for **flat dense**, **$$88.1\%$$** $$[84.9, 91.2]$$ for the **fixed-depth hierarchical baseline**, and **$$89.9\%$$** $$[86.7, 93.0]$$ for **Adaptive (heuristic)**. The paired-bootstrap delta of **Adaptive$$-$$Hierarchical** at $$k=10$$ is **$$+1.8$$** $$[-0.9, +4.7]$$. The large flat-to-hierarchical gain attributable to bidirectional diffusion with role-aware quotas **survives** the stricter split, while Adaptive remains **statistically tied** to the fixed-depth baseline.

LectureBank-Full Target-Disjoint R@10 Result (n=164): Diffusion Gain Survives, Adaptive Tied

On **LectureBank-Full**, the **hierarchical baseline** improves **token-level F$$_1$$ from $$1.9$$ to $$18.3$$** relative to **heuristic concatenation** of retrieved passages, but **lexical exact match remains $$0.0$$**. The large F$$_1$$ gain indicates that the hierarchical retriever feeds substantially more of the gold answer tokens into the generator, while the zero EM reflects that the generator's surface form does not match the gold reference verbatim. Because generation is treated only as a context-quality diagnostic, the F$$_1$$ improvement is read as evidence that the hierarchical context preserves more answerable evidence, not as a headline claim about generation quality.

LectureBank-Full Generation Diagnostic: Token-F1 1.9 → 18.3, EM Stays 0.0

Supplementary **error-taxonomy counts** for **LectureBank-Full** show that most residual retrieval misses under the paper's hierarchical and adaptive systems are **near-misses along the local prerequisite graph** — the retrieved concept is one or a few prerequisite-graph hops away from the gold target rather than being topically unrelated. Because residual errors cluster locally, the dataset offers little headroom for a depth-control fix and instead points toward **seed quality and graph noise** as the remaining bottlenecks.

LectureBank-Full Error Taxonomy: Residual Misses Are Near-Misses Along the Local Prerequisite Graph

LectureBank-Full Paired Delta: Adaptive vs Hierarchical Baseline = +0.7 [-2.1, +3.6]

Under a controlled token-cap protocol where **both Adaptive + token-aware** and **Hierarchical + token-aware** are reranked with the **same token-aware serialization policy**, the paired-bootstrap delta $$\Delta\text{R@}10 = $$ Adaptive − Hierarchical on **LectureBank-Full** ($$n = 143$$) is **negative and grows more negative as the cap is relaxed**: $$-0.9$$ $$[-2.4, +0.3]$$ at a 256-token cap ($$p = 0.191$$), $$-1.7$$ $$[-3.6, -0.1]$$ at 512 tokens ($$p = 0.045$$), and $$-3.4$$ $$[-6.2, -0.5]$$ at 768 tokens ($$p = 0.025$$). At the strictest 256-token cap the two systems are not individually distinguishable under the descriptive paired bootstrap; the individually significant negative deltas appear at the looser 512 and 768 caps. The reported $$p$$-values are descriptive paired-bootstrap tail probabilities and are **not adjusted across token caps**, so the trend is directional rather than a multiplicity-corrected significance result.

Token-Cap Comparison on LectureBank-Full: Adaptive Loses More as Cap Tightens

On **LectureBank-Full**, paired-bootstrap deltas between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** show that adaptive is most plausible in the **tight-budget region**. Averaged over $$k \in \{1, 2, 3, 4\}$$, the **mean $$\Delta\text{R@}k = +2.13$$ points**, with a 95% paired-bootstrap CI of $$[-0.64, +4.90]$$. Because the CI **includes zero**, the aggregate effect is **not statistically distinguishable from zero**; the tight-budget benefit is best read as a $$k=4$$ pointwise estimate rather than a regime-wide claim. The effect is also weaker than the diffusion-and-quota gain reported elsewhere in the paper.

LectureBank-Full Tight-Budget Advantage of Adaptive Depth Gating (Mean ΔR@k = +2.13 over k∈{1,2,3,4})

Within the tight-budget region on **LectureBank-Full**, the pointwise paired-bootstrap advantage of **Adaptive (heuristic) depth gating** over the **fixed-depth hierarchical baseline** **peaks at $$k = 4$$**, where $$\Delta\text{R@}4 = +6.4$$ points with a 95% CI of $$[1.0, 11.7]$$. Because the CI excludes zero, this is the one tight-budget $$k$$ value at which adaptive gating shows a statistically reliable pointwise gain on this benchmark.

LectureBank-Full ΔR@k Peaks at k=4 (+6.4 Points, CI [1.0, 11.7])

On **LectureBank-Full**, the paper's **deterministic bidirectional diffusion with role-aware quotas** lifts **Recall@$$10$$ by roughly $$18$$ points** relative to a **static parent-expansion** baseline (which expands each query along fixed prerequisite-parent links without diffusion). The same comparison shows that the **contrast-based gating term** of the adaptive controller adds **only about one R@$$10$$ point** on top of the diffusion-and-quota baseline (paired-bootstrap delta $$+1.05$$ pp, 95% CI $$[-1.52, +3.73]$$, $$p = 0.448$$). The Analysis section uses this contrast to conclude that the main retrieval gain comes from diffusion and quotas, not from depth gating.

LectureBank-Full Diffusion Gain over Static Parent Expansion (~18 R@10 Points)

Under this paper's strictest leakage control on prerequisite QA splits, only a small number of **unique held-out target concepts** remain after removing both train/test question-string reuse and target-concept overlap: **21 unique held-out targets on LectureBank-Full** and **18 on MOOC-CS**, distributed within just two question families. Because the surviving evidence is this narrow, the paper bounds its leakage-controlled headline conclusions to **curated, template-based prerequisite QA** rather than projecting them to open-domain retrieval.

Bounded Held-Out Targets After Strictest Leakage Control (21 LectureBank-Full, 18 MOOC-CS)

On **LectureBank-Full**, the strict-parity audit decomposes the retrieval gain by component: deterministic **bidirectional diffusion plus role-aware quotas** lifts R@10 by roughly **18 points** relative to static parent expansion, while the **contrast-gating** (adaptive-depth) component adds at most a nominal **+1.05 pp** in paired-bootstrap delta with 95% CI $$[-1.52, +3.73]$$ and $$p = 0.448$$. The contrast gate is therefore statistically tied with the fixed-depth hierarchical baseline, and the main retrieval gain is attributable to diffusion plus role-aware quotas rather than to adaptive depth.

LectureBank-Full Decomposition: Diffusion+Quotas Drive ~18 R@10 Points; Contrast Gating Adds At Most ~1 Point (Statistically Tied)

The **token-cap diagnostic** is the paper's first-use protocol for comparing two retrieval systems **after applying a shared token-aware serialization policy** to their returned contexts. Both systems serialize their retrieved items under the same rules and the same token budget, so any remaining difference can be attributed to **what was retrieved** rather than to **how it was packed into the prompt**. The diagnostic is analyzed **separately from the strict-parity headline comparisons**, which is what lets the paper keep traversal effects distinct from token-budget effects.

Token-Cap Diagnostic Under a Shared Serialization Policy

The **matched-budget best-first traversal comparator** is a graph-aware strict-parity baseline in which the prerequisite graph is traversed by expanding the highest-scoring frontier node first, subject to the same node/edge/token budget as the paper's diffusion retriever. Under the strict-parity contract, it shares the same encoder, $$m/k$$ budgets, split, and exact-ID matching rule as every other compared system, so any difference between this comparator and the paper's diffusion policy is attributable to *the graph policy itself* (best-first traversal versus dense-seeded local diffusion with role-aware quotas) rather than to budget or interface differences.

Matched-Budget Best-First Traversal Comparator in Strict-Parity Prerequisite Retrieval

On **MOOC-CS**, the paper additionally runs a configuration that uses a **multilingual encoder** plus a **CJK-only query rewrite** to handle Chinese-character queries. This configuration is treated as a **control experiment**, not as a headline strict-parity comparator: because it changes the encoder (and therefore the parity contract), its numbers are reported separately and are used only to diagnose how much of the MOOC-CS gap is attributable to language alignment rather than to graph policy.

Multilingual Encoder + CJK Query Rewrite as a MOOC-CS Control

The bidirectional-diffusion controller sets the ancestor depth $$d_\uparrow$$ and descendant depth $$d_\downarrow$$ per query as

$$d_\uparrow = \max\!\left\{1,\; d_\uparrow^{(0)} + \mathbf{1}[\ell(v^\star)\ge\tau_{\text{high}}] - \mathbf{1}[\ell(v^\star)\le\tau_{\text{low}}] + \mathbf{1}[\Delta>\gamma_{\text{high}}]\right\},$$

$$d_\downarrow = \max\!\left\{0,\; d_\downarrow^{(0)} + \mathbf{1}[\ell(v^\star)\le\tau_{\text{low}}] - \mathbf{1}[\ell(v^\star)\ge\tau_{\text{high}}] + \mathbf{1}[\Delta<\gamma_{\text{low}}]\right\},$$

where $$v^\star$$ is the highest-scoring dense seed, $$\ell(v^\star)$$ is its hierarchy level, and $$\Delta = s_1 - s_5$$ is the top-vs-fifth dense-score contrast. A high seed level or large $$\Delta$$ increases upward depth; a low seed level or small $$\Delta$$ increases downward depth, with floors of $$1$$ and $$0$$.

Adaptive Ancestor/Descendant Depth Controller Formula

On the **QASC** validation split ($$n = 926$$), the **paired delta** between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** in R@$$10$$ is only **$$+0.5$$ points**, with a 95% paired-bootstrap CI of **$$[-0.5, +1.5]$$**. Because the CI straddles zero, the two systems are statistically **tied** under strict parity, which is why the paper presents **adaptive depth gating as an optional extension rather than the core result** on this dataset.

QASC Paired Delta: Adaptive vs Hierarchical Baseline = +0.5 [-0.5, +1.5]

Under the stricter **target-concept-disjoint** leakage control, **Adaptive (heuristic)** depth gating remains **statistically tied** to the **fixed-depth hierarchical baseline** on both curated prerequisite benchmarks: the paired Adaptive$$-$$Hierarchical delta at $$k=10$$ is **$$+1.8$$** $$[-0.9, +4.7]$$ on **LectureBank-Full** and **$$-1.2$$** $$[-2.9, +0.4]$$ on **MOOC-CS** (each CI straddles zero). Adaptive depth gating therefore does not produce a reliable improvement over fixed-depth traversal once interfaces and splits are held constant, even under the stricter leakage audit.

Adaptive Depth Gating Stays Tied to Hierarchical Baseline Under Target-Disjoint Control

The paper's gate-only ablation **fixes diffusion and role-aware quotas** and toggles only the **contrast term** of the adaptive controller, so the comparison isolates the contribution of the dense-score contrast $$\Delta = s_1 - s_5$$ on top of the level-only gate. The ablation rows therefore vary **only** the contrast indicator in the controller; every other component of the retrieval pipeline (diffusion rule, quotas, encoder, $$m$$, $$k$$, split, matching rule) is held constant.

Learn Before

Related