Within the tight-budget region on **LectureBank-Full**, the pointwise paired-bootstrap advantage of **Adaptive (heuristic) depth gating** over the **fixed-depth hierarchical baseline** **peaks at $$k = 4$$**, where $$\Delta\text{R@}4 = +6.4$$ points with a 95% CI of $$[1.0, 11.7]$$. Because the CI excludes zero, this is the one tight-budget $$k$$ value at which adaptive gating shows a statistically reliable pointwise gain on this benchmark.

Google

**LectureBank-Full** is the expanded configuration of the LectureBank corpus used as a prerequisite-QA / retrieval benchmark. It contains $$1{,}717$$ English NLP-related lecture slide files and manual concept-pair prerequisite annotations over $$322$$ topics. The task is to retrieve, for a given target concept (query), the set of gold prerequisite concepts from a fixed candidate pool. Systems are scored with ranking metrics such as Recall@$$k$$ (R@$$k$$); R@$$10$$ measures the fraction of gold prerequisites recovered in the top-$$10$$ retrieved candidates. Standard splits hold out concept pairs for evaluation, and headline comparisons fix the encoder, candidate pool, cutoff, matching rule, and split policy.

LectureBank-Full Prerequisite QA Benchmark

This organizer keeps part 2 of the method details for Adaptive Depth Gating for Prerequisite Retrieval small enough for learners to study without a long flat child list.

Method Part 2: Adaptive Depth Gating for Prerequisite Retrieval (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups the paper content from Results > Tight Budgets and Token Caps.

Tight Budgets And Token Caps (Results) in Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

On the **LectureBank-Full** prerequisite-QA benchmark under the paper's strict-parity contract, **flat dense** retrieval reaches **R@$$10 = 55.5$$** (95% CI $$[51.9, 59.3]$$), a **fixed-depth hierarchical baseline** reaches **$$86.0$$** $$[81.9, 89.7]$$, and **Adaptive (heuristic) depth gating** reaches **$$86.7$$** $$[82.9, 90.6]$$. Because the encoder, candidate pool, cutoff $$k = 10$$, matching rule, and split are held constant, the $$\approx 30$$-point jump from $$55.5$$ to $$86.0$$ is attributable to **deterministic bidirectional diffusion with role-aware quotas**, not to adaptive depth control.

LectureBank-Full R@10 Gain from Diffusion and Role-Aware Quotas

In this paper, **LectureBank-Full** is consumed as a prerequisite graph with **$$208$$ concepts**, **$$899$$ prerequisite edges**, and **$$1{,}421$$ QA pairs** derived from instructional materials. Graph descriptors computed from the released graph and concept texts are: mean depth $$3.2$$, fraction of nodes with depth $$\geq 1$$ equal to $$90.4\%$$, mean degree $$8.6$$, and mean concept-text length $$66.3$$ tokens. The dataset is one of the two curated prerequisite benchmarks used for headline retrieval comparisons.

LectureBank-Full Configuration Used in Hierarchical Prerequisite RAG (208 Concepts, 899 Edges, 1,421 QA)

Under the **target-concept-disjoint** control on **LectureBank-Full** ($$n = 164$$), R@$$10$$ is **$$56.9\%$$** $$[53.9, 60.1]$$ for **flat dense**, **$$88.1\%$$** $$[84.9, 91.2]$$ for the **fixed-depth hierarchical baseline**, and **$$89.9\%$$** $$[86.7, 93.0]$$ for **Adaptive (heuristic)**. The paired-bootstrap delta of **Adaptive$$-$$Hierarchical** at $$k=10$$ is **$$+1.8$$** $$[-0.9, +4.7]$$. The large flat-to-hierarchical gain attributable to bidirectional diffusion with role-aware quotas **survives** the stricter split, while Adaptive remains **statistically tied** to the fixed-depth baseline.

LectureBank-Full Target-Disjoint R@10 Result (n=164): Diffusion Gain Survives, Adaptive Tied

On **LectureBank-Full**, the **hierarchical baseline** improves **token-level F$$_1$$ from $$1.9$$ to $$18.3$$** relative to **heuristic concatenation** of retrieved passages, but **lexical exact match remains $$0.0$$**. The large F$$_1$$ gain indicates that the hierarchical retriever feeds substantially more of the gold answer tokens into the generator, while the zero EM reflects that the generator's surface form does not match the gold reference verbatim. Because generation is treated only as a context-quality diagnostic, the F$$_1$$ improvement is read as evidence that the hierarchical context preserves more answerable evidence, not as a headline claim about generation quality.

LectureBank-Full Generation Diagnostic: Token-F1 1.9 → 18.3, EM Stays 0.0

Supplementary **error-taxonomy counts** for **LectureBank-Full** show that most residual retrieval misses under the paper's hierarchical and adaptive systems are **near-misses along the local prerequisite graph** — the retrieved concept is one or a few prerequisite-graph hops away from the gold target rather than being topically unrelated. Because residual errors cluster locally, the dataset offers little headroom for a depth-control fix and instead points toward **seed quality and graph noise** as the remaining bottlenecks.

LectureBank-Full Error Taxonomy: Residual Misses Are Near-Misses Along the Local Prerequisite Graph

On **LectureBank-Full**, the **paired delta** between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** in R@$$10$$ is only **$$+0.7$$ points**, with a 95% paired-bootstrap CI of **$$[-2.1, +3.6]$$**. Because the CI straddles zero, the two systems are statistically **tied** under strict parity, which is why the paper treats **depth gating as optional rather than essential** on this benchmark.

LectureBank-Full Paired Delta: Adaptive vs Hierarchical Baseline = +0.7 [-2.1, +3.6]

Under a controlled token-cap protocol where **both Adaptive + token-aware** and **Hierarchical + token-aware** are reranked with the **same token-aware serialization policy**, the paired-bootstrap delta $$\Delta\text{R@}10 = $$ Adaptive − Hierarchical on **LectureBank-Full** ($$n = 143$$) is **negative and grows more negative as the cap is relaxed**: $$-0.9$$ $$[-2.4, +0.3]$$ at a 256-token cap ($$p = 0.191$$), $$-1.7$$ $$[-3.6, -0.1]$$ at 512 tokens ($$p = 0.045$$), and $$-3.4$$ $$[-6.2, -0.5]$$ at 768 tokens ($$p = 0.025$$). At the strictest 256-token cap the two systems are not individually distinguishable under the descriptive paired bootstrap; the individually significant negative deltas appear at the looser 512 and 768 caps. The reported $$p$$-values are descriptive paired-bootstrap tail probabilities and are **not adjusted across token caps**, so the trend is directional rather than a multiplicity-corrected significance result.

Token-Cap Comparison on LectureBank-Full: Adaptive Loses More as Cap Tightens

On **LectureBank-Full**, paired-bootstrap deltas between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** show that adaptive is most plausible in the **tight-budget region**. Averaged over $$k \in \{1, 2, 3, 4\}$$, the **mean $$\Delta\text{R@}k = +2.13$$ points**, with a 95% paired-bootstrap CI of $$[-0.64, +4.90]$$. Because the CI **includes zero**, the aggregate effect is **not statistically distinguishable from zero**; the tight-budget benefit is best read as a $$k=4$$ pointwise estimate rather than a regime-wide claim. The effect is also weaker than the diffusion-and-quota gain reported elsewhere in the paper.

LectureBank-Full Tight-Budget Advantage of Adaptive Depth Gating (Mean ΔR@k = +2.13 over k∈{1,2,3,4})

LectureBank-Full ΔR@k Peaks at k=4 (+6.4 Points, CI [1.0, 11.7])

On **LectureBank-Full**, the paper's **deterministic bidirectional diffusion with role-aware quotas** lifts **Recall@$$10$$ by roughly $$18$$ points** relative to a **static parent-expansion** baseline (which expands each query along fixed prerequisite-parent links without diffusion). The same comparison shows that the **contrast-based gating term** of the adaptive controller adds **only about one R@$$10$$ point** on top of the diffusion-and-quota baseline (paired-bootstrap delta $$+1.05$$ pp, 95% CI $$[-1.52, +3.73]$$, $$p = 0.448$$). The Analysis section uses this contrast to conclude that the main retrieval gain comes from diffusion and quotas, not from depth gating.

LectureBank-Full Diffusion Gain over Static Parent Expansion (~18 R@10 Points)

Under this paper's strictest leakage control on prerequisite QA splits, only a small number of **unique held-out target concepts** remain after removing both train/test question-string reuse and target-concept overlap: **21 unique held-out targets on LectureBank-Full** and **18 on MOOC-CS**, distributed within just two question families. Because the surviving evidence is this narrow, the paper bounds its leakage-controlled headline conclusions to **curated, template-based prerequisite QA** rather than projecting them to open-domain retrieval.

Bounded Held-Out Targets After Strictest Leakage Control (21 LectureBank-Full, 18 MOOC-CS)

On **LectureBank-Full**, the strict-parity audit decomposes the retrieval gain by component: deterministic **bidirectional diffusion plus role-aware quotas** lifts R@10 by roughly **18 points** relative to static parent expansion, while the **contrast-gating** (adaptive-depth) component adds at most a nominal **+1.05 pp** in paired-bootstrap delta with 95% CI $$[-1.52, +3.73]$$ and $$p = 0.448$$. The contrast gate is therefore statistically tied with the fixed-depth hierarchical baseline, and the main retrieval gain is attributable to diffusion plus role-aware quotas rather than to adaptive depth.

LectureBank-Full Decomposition: Diffusion+Quotas Drive ~18 R@10 Points; Contrast Gating Adds At Most ~1 Point (Statistically Tied)

The paper's main bidirectional-diffusion experiments fix the following hyperparameters. Base traversal depths: $$d_\uparrow^{(0)} = 6$$ (ancestors) and $$d_\downarrow^{(0)} = 2$$ (descendants). Adaptive-controller thresholds on the top seed's hierarchy level: $$(\tau_{\text{low}}, \tau_{\text{high}}) = (2, 5)$$. Adaptive-controller thresholds on the dense-score contrast $$\Delta = s_1 - s_5$$: $$(\gamma_{\text{low}}, \gamma_{\text{high}}) = (0.05, 0.15)$$. Per-hop penalties: $$\delta = 0.03$$ (upward) and $$\delta_{\text{child}} = 0.025$$ (downward). The asymmetries $$d_\uparrow^{(0)} > d_\downarrow^{(0)}$$ and $$\delta > \delta_{\text{child}}$$ bias the policy toward deeper but more strongly hop-penalized ancestor traversal than descendant traversal.

Bidirectional Diffusion Hyperparameter Settings (LectureBank-Full Main Runs)

On **MOOC-CS**, the paired-bootstrap $$\Delta\text{R@}k$$ curve between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** **stays near zero throughout** $$k$$, in contrast to LectureBank-Full where adaptive shows a pointwise advantage at $$k=4$$. Even where the LectureBank-Full pointwise effect appears, it is weaker than the diffusion-and-quota gain. This reading supports the paper's descriptive conclusion that adaptive depth gating's tight-budget benefit is **dataset-specific and not regime-wide**.

MOOC-CS ΔR@k Curve Stays Near Zero Across k (Adaptive vs Hierarchical)

On **MOOC-CS** under the paper's default English MiniLM strict-parity configuration, the headline retrieval table shows two clusters in R@$$10$$: **LightRAG-style** and **truncated personalized PageRank** baselines attain the **best R@$$10$$** in the range **$$25.6$$-$$25.9$$**, while **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** are **tied at $$23.1$$**. The paper labels this as its **second corrective point**: under an English-only encoder, graph traversal-policy choices do not separate adaptive depth from the hierarchical baseline, and the better-performing systems are the LightRAG-style and PPR-style baselines rather than the paper's hierarchical traversal — which motivates the subsequent language-matched controls reported in the MOOC-CS cleaning-controls table.

MOOC-CS Headline R@10 Numbers: LightRAG/Truncated-PPR 25.6-25.9 vs Adaptive/Hierarchical Tied at 23.1

The paper's token-budget guidance is a **dataset-aware conditional recommendation**: **adaptive retrieval can be useful when $$k$$ is small and token caps are loose** (where its tight-budget pointwise gain on LectureBank-Full appears at $$k=4$$); **under strict token caps, the simpler fixed-depth hierarchical baseline is the safer descriptive default** on LectureBank-Full, because adaptive contexts are longer on average and lose more under truncation. The recommendation is dataset-aware: it is supported on LectureBank-Full and is not supported on MOOC-CS, where the $$\Delta\text{R@}k$$ curve stays near zero. It is framed as **descriptive guidance** rather than a multiplicity-corrected significance claim.

Learn Before

Related