On the **LectureBank-Full** prerequisite-QA benchmark under the paper's strict-parity contract, **flat dense** retrieval reaches **R@$$10 = 55.5$$** (95% CI $$[51.9, 59.3]$$), a **fixed-depth hierarchical baseline** reaches **$$86.0$$** $$[81.9, 89.7]$$, and **Adaptive (heuristic) depth gating** reaches **$$86.7$$** $$[82.9, 90.6]$$. Because the encoder, candidate pool, cutoff $$k = 10$$, matching rule, and split are held constant, the $$\approx 30$$-point jump from $$55.5$$ to $$86.0$$ is attributable to **deterministic bidirectional diffusion with role-aware quotas**, not to adaptive depth control.

Google

**LectureBank-Full** is the expanded configuration of the LectureBank corpus used as a prerequisite-QA / retrieval benchmark. It contains $$1{,}717$$ English NLP-related lecture slide files and manual concept-pair prerequisite annotations over $$322$$ topics. The task is to retrieve, for a given target concept (query), the set of gold prerequisite concepts from a fixed candidate pool. Systems are scored with ranking metrics such as Recall@$$k$$ (R@$$k$$); R@$$10$$ measures the fraction of gold prerequisites recovered in the top-$$10$$ retrieved candidates. Standard splits hold out concept pairs for evaluation, and headline comparisons fix the encoder, candidate pool, cutoff, matching rule, and split policy.

LectureBank-Full Prerequisite QA Benchmark

This organizer keeps part 3 of the method details for Strict-Parity Evaluation Contract for Graph-Aware small enough for learners to study without a long flat child list.

Method Part 3: Strict-Parity Evaluation Contract for Graph-Aware (Auditable Strict-Parity Graph-RAG Paper)

This organizer keeps part 2 of the method details for Bidirectional Prerequisite Diffusion with Role-Aware small enough for learners to study without a long flat child list.

Method Part 2: Bidirectional Prerequisite Diffusion with Role-Aware (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups metrics, datasets, statistical protocol, and result evidence used by LectureBank-Full: Diffusion Plus Quotas Account For The Main Gain.

Evaluation Evidence for LectureBank-Full: Diffusion Plus Quotas Account For The Main Gain (Auditable Strict-Parity Graph-RAG Paper)

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

LectureBank-Full R@10 Gain from Diffusion and Role-Aware Quotas

In this paper, **LectureBank-Full** is consumed as a prerequisite graph with **$$208$$ concepts**, **$$899$$ prerequisite edges**, and **$$1{,}421$$ QA pairs** derived from instructional materials. Graph descriptors computed from the released graph and concept texts are: mean depth $$3.2$$, fraction of nodes with depth $$\geq 1$$ equal to $$90.4\%$$, mean degree $$8.6$$, and mean concept-text length $$66.3$$ tokens. The dataset is one of the two curated prerequisite benchmarks used for headline retrieval comparisons.

LectureBank-Full Configuration Used in Hierarchical Prerequisite RAG (208 Concepts, 899 Edges, 1,421 QA)

Under the **target-concept-disjoint** control on **LectureBank-Full** ($$n = 164$$), R@$$10$$ is **$$56.9\%$$** $$[53.9, 60.1]$$ for **flat dense**, **$$88.1\%$$** $$[84.9, 91.2]$$ for the **fixed-depth hierarchical baseline**, and **$$89.9\%$$** $$[86.7, 93.0]$$ for **Adaptive (heuristic)**. The paired-bootstrap delta of **Adaptive$$-$$Hierarchical** at $$k=10$$ is **$$+1.8$$** $$[-0.9, +4.7]$$. The large flat-to-hierarchical gain attributable to bidirectional diffusion with role-aware quotas **survives** the stricter split, while Adaptive remains **statistically tied** to the fixed-depth baseline.

LectureBank-Full Target-Disjoint R@10 Result (n=164): Diffusion Gain Survives, Adaptive Tied

On **LectureBank-Full**, the **hierarchical baseline** improves **token-level F$$_1$$ from $$1.9$$ to $$18.3$$** relative to **heuristic concatenation** of retrieved passages, but **lexical exact match remains $$0.0$$**. The large F$$_1$$ gain indicates that the hierarchical retriever feeds substantially more of the gold answer tokens into the generator, while the zero EM reflects that the generator's surface form does not match the gold reference verbatim. Because generation is treated only as a context-quality diagnostic, the F$$_1$$ improvement is read as evidence that the hierarchical context preserves more answerable evidence, not as a headline claim about generation quality.

LectureBank-Full Generation Diagnostic: Token-F1 1.9 → 18.3, EM Stays 0.0

Supplementary **error-taxonomy counts** for **LectureBank-Full** show that most residual retrieval misses under the paper's hierarchical and adaptive systems are **near-misses along the local prerequisite graph** — the retrieved concept is one or a few prerequisite-graph hops away from the gold target rather than being topically unrelated. Because residual errors cluster locally, the dataset offers little headroom for a depth-control fix and instead points toward **seed quality and graph noise** as the remaining bottlenecks.

LectureBank-Full Error Taxonomy: Residual Misses Are Near-Misses Along the Local Prerequisite Graph

On **LectureBank-Full**, the **paired delta** between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** in R@$$10$$ is only **$$+0.7$$ points**, with a 95% paired-bootstrap CI of **$$[-2.1, +3.6]$$**. Because the CI straddles zero, the two systems are statistically **tied** under strict parity, which is why the paper treats **depth gating as optional rather than essential** on this benchmark.

LectureBank-Full Paired Delta: Adaptive vs Hierarchical Baseline = +0.7 [-2.1, +3.6]

Under a controlled token-cap protocol where **both Adaptive + token-aware** and **Hierarchical + token-aware** are reranked with the **same token-aware serialization policy**, the paired-bootstrap delta $$\Delta\text{R@}10 = $$ Adaptive − Hierarchical on **LectureBank-Full** ($$n = 143$$) is **negative and grows more negative as the cap is relaxed**: $$-0.9$$ $$[-2.4, +0.3]$$ at a 256-token cap ($$p = 0.191$$), $$-1.7$$ $$[-3.6, -0.1]$$ at 512 tokens ($$p = 0.045$$), and $$-3.4$$ $$[-6.2, -0.5]$$ at 768 tokens ($$p = 0.025$$). At the strictest 256-token cap the two systems are not individually distinguishable under the descriptive paired bootstrap; the individually significant negative deltas appear at the looser 512 and 768 caps. The reported $$p$$-values are descriptive paired-bootstrap tail probabilities and are **not adjusted across token caps**, so the trend is directional rather than a multiplicity-corrected significance result.

Token-Cap Comparison on LectureBank-Full: Adaptive Loses More as Cap Tightens

On **LectureBank-Full**, paired-bootstrap deltas between **Adaptive (heuristic) depth gating** and the **fixed-depth hierarchical baseline** show that adaptive is most plausible in the **tight-budget region**. Averaged over $$k \in \{1, 2, 3, 4\}$$, the **mean $$\Delta\text{R@}k = +2.13$$ points**, with a 95% paired-bootstrap CI of $$[-0.64, +4.90]$$. Because the CI **includes zero**, the aggregate effect is **not statistically distinguishable from zero**; the tight-budget benefit is best read as a $$k=4$$ pointwise estimate rather than a regime-wide claim. The effect is also weaker than the diffusion-and-quota gain reported elsewhere in the paper.

LectureBank-Full Tight-Budget Advantage of Adaptive Depth Gating (Mean ΔR@k = +2.13 over k∈{1,2,3,4})

Within the tight-budget region on **LectureBank-Full**, the pointwise paired-bootstrap advantage of **Adaptive (heuristic) depth gating** over the **fixed-depth hierarchical baseline** **peaks at $$k = 4$$**, where $$\Delta\text{R@}4 = +6.4$$ points with a 95% CI of $$[1.0, 11.7]$$. Because the CI excludes zero, this is the one tight-budget $$k$$ value at which adaptive gating shows a statistically reliable pointwise gain on this benchmark.

LectureBank-Full ΔR@k Peaks at k=4 (+6.4 Points, CI [1.0, 11.7])

On **LectureBank-Full**, the paper's **deterministic bidirectional diffusion with role-aware quotas** lifts **Recall@$$10$$ by roughly $$18$$ points** relative to a **static parent-expansion** baseline (which expands each query along fixed prerequisite-parent links without diffusion). The same comparison shows that the **contrast-based gating term** of the adaptive controller adds **only about one R@$$10$$ point** on top of the diffusion-and-quota baseline (paired-bootstrap delta $$+1.05$$ pp, 95% CI $$[-1.52, +3.73]$$, $$p = 0.448$$). The Analysis section uses this contrast to conclude that the main retrieval gain comes from diffusion and quotas, not from depth gating.

LectureBank-Full Diffusion Gain over Static Parent Expansion (~18 R@10 Points)

Under this paper's strictest leakage control on prerequisite QA splits, only a small number of **unique held-out target concepts** remain after removing both train/test question-string reuse and target-concept overlap: **21 unique held-out targets on LectureBank-Full** and **18 on MOOC-CS**, distributed within just two question families. Because the surviving evidence is this narrow, the paper bounds its leakage-controlled headline conclusions to **curated, template-based prerequisite QA** rather than projecting them to open-domain retrieval.

Bounded Held-Out Targets After Strictest Leakage Control (21 LectureBank-Full, 18 MOOC-CS)

On **LectureBank-Full**, the strict-parity audit decomposes the retrieval gain by component: deterministic **bidirectional diffusion plus role-aware quotas** lifts R@10 by roughly **18 points** relative to static parent expansion, while the **contrast-gating** (adaptive-depth) component adds at most a nominal **+1.05 pp** in paired-bootstrap delta with 95% CI $$[-1.52, +3.73]$$ and $$p = 0.448$$. The contrast gate is therefore statistically tied with the fixed-depth hierarchical baseline, and the main retrieval gain is attributable to diffusion plus role-aware quotas rather than to adaptive depth.

LectureBank-Full Decomposition: Diffusion+Quotas Drive ~18 R@10 Points; Contrast Gating Adds At Most ~1 Point (Statistically Tied)

**LightRAG-style retrieval** is included as a graph-aware comparator in this paper's strict-parity baseline set. Following the LightRAG framework, candidate concepts are scored using a dual-level entity-and-relation traversal combined with embedding similarity over the prerequisite graph. Under the strict-parity contract, the baseline shares the same encoder, $$m/k$$ budgets, split, and exact-ID matching rule as the other compared systems, so it isolates LightRAG-style traversal as the graph policy.

LightRAG-Style Baseline in Strict-Parity Prerequisite Retrieval

**KAG-style structural retrieval** is included as a graph-aware comparator in this paper's strict-parity baseline set. Following the KAG framework, candidate concepts are scored by structural reasoning over the prerequisite graph together with the dense encoder over the shared seed pool. Under the strict-parity contract, the baseline shares the same encoder, $$m/k$$ budgets, split, and exact-ID matching rule as the other compared systems, so it isolates KAG-style structural retrieval as the graph policy.

KAG-Style Structural Baseline in Strict-Parity Prerequisite Retrieval

On the **MOOC-CS** prerequisite benchmark, graph-aware retrieval gains under the default **English-only** setup are **limited**, and become **substantial only after** introducing **language-matched query and encoder controls** (matching the language of the query and the encoder to the underlying course materials). The result is reported under the paper's strict-parity contract, which fixes the candidate pool, cutoff $$k$$, matching rule, and split policy and varies only the language-matching condition, so the change in graph gain is attributable to language matching rather than to graph policy. The finding illustrates that graph-specific gains can be masked or inflated by interface choices that are unrelated to graph structure.

MOOC-CS Graph Gain Requires Language-Matched Controls

**SP+** is the paper's first-use extension of the strict-parity (SP) contract that **additionally allows a learned reranker** on top of the matched encoder, dense candidate pool, cutoff, split, and matching rule. SP+ results are **reported separately** from SP so that any reranker-driven gain is not conflated with the graph-traversal effect that SP is designed to isolate. In practice, SP fixes the retrieval interface across compared systems, and SP+ layers a reranker onto that same interface as an explicitly labeled diagnostic.

SP+ Strict Parity Plus a Learned Reranker (Reported Separately)

The Analysis-section **operational lesson** drawn from MOOC-CS is **not** that adaptive gating always helps, but rather that **language-matched seeding is a prerequisite to benefiting from graph expansion**. Concretely, on MOOC-CS the contrast gate adds almost nothing on top of diffusion because the **dense seed pool is already weakened by bilingual aliasing and sparse/noisy edges**, so graph traversal has no high-quality seeds to expand from. The lesson reframes graph-RAG advice for prerequisite retrieval: invest in matching the encoder/query language to the corpus **before** attributing wins or losses to traversal-policy choices.

Language-Matched Seeding as a Prerequisite for Graph-Expansion Gains

In the bidirectional-diffusion retriever, for a path $$p$$ from a dense seed $$u$$ to a target node $$v$$, the upward and downward path scores are

$$s_\uparrow(p) = s_0(u) - \delta\,|p|,\qquad s_\downarrow(p) = s_0(u) - \delta_{\text{child}}\,|p|,$$

where $$s_0(u)$$ is the seed's dense score, $$|p|$$ is the path length in edges, and $$\delta$$ (upward) and $$\delta_{\text{child}}$$ (downward) are role-specific additive hop penalties. The scores are linear in path length, so each additional hop subtracts a fixed amount from the seed's dense score.

Hop-Penalized Path Score in Bidirectional Diffusion

The final score of a candidate concept node $$v$$ under bidirectional diffusion is the **maximum** over its own dense score and the best upward and downward reaching paths:

$$S(v) = \max\!\left\{\, s_0(v),\; \max_{p\in\mathcal{P}_\uparrow(v)} s_\uparrow(p),\; \max_{p\in\mathcal{P}_\downarrow(v)} s_\downarrow(p)\,\right\},$$

where $$\mathcal{P}_\uparrow(v)$$ and $$\mathcal{P}_\downarrow(v)$$ are the truncated ancestor and descendant path sets from dense seeds. Using $$\max$$ rather than summing (as in random-walk propagation) keeps the score equal to the strongest single justification for $$v$$ — either being a strong seed itself or being reachable via a short, low-penalty path from one.

Learn Before

Related