This organizer groups method, retrieval, ranking, and graph-structure details used by Claim-Level Traceability in Graph-RAG Evaluation.

Method Details for Claim-Level Traceability in Graph-RAG Evaluation (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups metrics, datasets, statistical protocol, and result evidence used by Claim-Level Traceability in Graph-RAG Evaluation.

Evaluation Evidence for Claim-Level Traceability in Graph-RAG Evaluation (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups scope, caveats, licensing, ethical, and interpretation notes connected to Claim-Level Traceability in Graph-RAG Evaluation.

Scope Notes for Claim-Level Traceability in Graph-RAG Evaluation (Auditable Strict-Parity Graph-RAG Paper)

**Claim-level traceability** is a reporting discipline in which every empirical claim in a graph-RAG evaluation is linked to the **specific artifact** that produced it — the exact split, candidate pool, encoder, cutoff, matching rule, and configuration — so that an external reader can reproduce the number behind the claim. The paper operationalizes this discipline through two concrete artifacts named in the Budgeted context construction and reproducibility paragraph: a **claim-to-artifact traceability matrix** that maps each empirical claim to the run that produced it, and **source-derived paper-of-record table fragments** that are generated directly from the underlying experiment artifacts rather than transcribed by hand. The discipline operationalizes the broader case for **reproducible IR benchmarking** (reference implementations, show-your-work reporting, split design, controlled benchmark assumptions) and for **auditable-artifact reporting standards** (data statements, datasheets, model cards, reproducibility checklists) at the level of individual paper claims.

Google

**Reproducible IR benchmarking and evaluation-variance control** is a methodological tradition in information retrieval and adjacent ML evaluation that treats **how a result is produced and reported** as part of the result itself. Four influential strands define this tradition. (i) **Reproducible reference implementations and statistical analyses on shared benchmarks**: Kamalloo et al. (SIGIR 2024) release reference dense and sparse retrievers on **BEIR**, together with effect-size meta-analyses and an official leaderboard, so that compared systems share the same artifacts and statistical conventions. (ii) **Improved reporting of experimental results**: Dodge et al. (EMNLP 2019) show that test-set numbers alone are insufficient and prescribe reporting **validation performance as a function of compute budget**, since conclusions about which model is best can flip with hyperparameter search effort. (iii) **Split design as a controlled variable**: Gorman and Bedrick (ACL 2019) show that single **standard train/test splits** can produce **unstable system rankings**, and argue for **randomized splits** to test whether reported gains are reproducible across resamplings. (iv) **Hidden benchmark assumptions**: Dehghani et al. (2021) formalize the **benchmark lottery**, demonstrating that the choice of benchmark tasks (and other unstated reporting choices) can flip the apparent ordering of algorithms. Together these works establish that retrieval and ML claims must be reported with enough artifact, split, and configuration detail for variance and benchmark choices to be audited — the reporting line that downstream claim-level traceability discipline extends.

Reproducible IR Benchmarking and Evaluation-Variance Control

**Auditable-artifact reporting standards** are a tradition in machine learning, NLP, and adjacent fields that attaches **standardized disclosure documents** to the artifacts a research community produces — datasets, trained models, and paper submissions — so that downstream users can independently audit how each artifact was made, what it contains, and what claims it can support. Four founding works define the tradition: (i) **data statements** (Bender and Friedman, TACL 2018), structured documents attached to NLP datasets that disclose curation rationale, language variety, and speaker and annotator demographics; (ii) **datasheets for datasets** (Gebru et al., CACM 2021), a 57-question form covering motivation, composition, collection, preprocessing, uses, distribution, and maintenance of any dataset; (iii) **model cards for model reporting** (Mitchell et al., FAT* 2019), short documents accompanying trained models that report intended use, factors, evaluation and training data, subgroup-disaggregated metrics, and ethical considerations; and (iv) **the NeurIPS 2019 reproducibility program and ML Reproducibility Checklist** (Pineau et al., JMLR 2021), which attaches an explicit reproducibility checklist to conference submissions and pairs it with a code-submission policy. The shared discipline across all four is that **each artifact ships with the documentation a third party needs to audit it**, rather than relying on prose buried inside the paper. This tradition is methodologically distinct from the IR-evaluation reproducibility line (reference implementations, statistical controls, split-design hygiene): it is about **per-artifact disclosure** rather than benchmark-level evaluation variance.

Auditable-Artifact Reporting Standards: Data Statements, Datasheets, Model Cards, Reproducibility Checklists

This organizer keeps part 1 of the scope details for Abstract small enough for learners to study without a long flat child list.

Scope Part 1: Abstract (Auditable Strict-Parity Graph-RAG Paper)

This organizer groups the paper content from Related Work > Budgeted context construction and reproducibility..

Budgeted Context Construction And Reproducibility (Related Work) in Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls

Claim-Level Traceability in Graph-RAG Evaluation

The paper recommends its hierarchical prerequisite graph-RAG systems be used **only as assistive tools for instructors or content curators**, and explicitly **not as unsupervised tutors**. The recommendation is motivated by a **pedagogical risk**: incorrect prerequisite chains or low-faithfulness generations could mislead learners if presented without expert oversight. The scope restriction is the paper's primary ethical control on downstream use.

Assistive-Tool-Only Deployment Recommendation for Hierarchical Prerequisite Graph RAG

DEPRECATED — duplicate of "Dataset Licenses, Redistribution, and Instructor Attribution Deferred to Released Data Cards" (hk6XHIzuLTQAfe4hXmtC), which states the same atomic idea drawn from the same Ethical Considerations sentence. Both nodes are paper-derived children of the paper node and of the Auditable-Artifact Reporting Standards node, both cite the same span, and both encode the claim that licenses, redistribution rights, and instructor-attribution constraints for MOOC-derived materials are documented in the released data cards rather than resolved in the paper text. Prefer hk6XHIzuLTQAfe4hXmtC because its title and content more faithfully reproduce the paper's wording ("deferred to released data cards") and explicitly enumerate redistribution rights, which the f2iITDAe91FFGfCUGQOa node omits.

Dataset License and Attribution Disclosure via Released Data Cards

The paper does **not** assert that **non-privacy ethical dimensions are zero**. Specifically, **dataset licenses**, **redistribution rights**, and **instructor-attribution constraints** for MOOC-derived materials are **documented in the released data cards** rather than claimed as resolved in the paper body. This is a deliberate scoping move: the paper localizes auditable ethical commitments to the data-card artifact instead of the manuscript prose.

Non-Privacy Ethical Dimensions Deferred to Released Data Cards (Licenses, Redistribution, Instructor Attribution)

**Curated prerequisite DAGs** over educational concepts are used in this paper as a graph-RAG **test bed** rather than a target application. Their value is that the directed acyclic graph (DAG) of prerequisite relations is **explicit, inspectable, and auditable**, which makes graph expansion policies easy to specify and to ablate. The aim is to test whether graph-aware retrieval helps **beyond a matched dense baseline**, not to argue that graph retrieval dominates general RAG settings. Consequently, conclusions drawn on these DAGs are scoped to curated, template-based prerequisite QA.

Prerequisite DAGs as a Graph-RAG Evaluation Test Bed

**Adaptive depth gating** is a per-query controller that sets the ancestor traversal depth $$d_\uparrow$$ and descendant traversal depth $$d_\downarrow$$ as a function of the **hierarchy level $$\ell(v^\star)$$** of the top dense seed and the **dense-score contrast** $$\Delta = s_1 - s_5$$ between the top and fifth dense scores. From base depths $$d_\uparrow^{(0)}$$ and $$d_\downarrow^{(0)}$$, the controller adds an indicator when $$\ell(v^\star)$$ is high (pushing $$d_\uparrow$$ up) or low (pushing $$d_\downarrow$$ up), and adds an indicator when $$\Delta$$ is large (more upward depth) or small (more downward depth), with floors of $$1$$ and $$0$$ respectively. Across the studied benchmarks (LectureBank-Full, QASC, HotpotQA boundary probe), adaptive gating remains **statistically tied** to a fixed-depth hierarchical baseline once interfaces and splits are matched, so it is treated as an optional extension rather than a core claim.

Adaptive Depth Gating for Prerequisite Retrieval

**Question-disjoint** and **target-concept-disjoint** controls are the two leakage-auditing splits used in this paper for canonical prerequisite-QA benchmarks. The audit is motivated by the observation that canonical splits reuse many **exact question strings** and **target concepts** across train and test, so a system can match memorized surface forms rather than retrieve true prerequisites. Under the **question-disjoint** control, no test question string appears in training. Under the **target-concept-disjoint** control, no test target concept appears as a target in training. Results obtained only under these controls are treated as the **trustworthy** evidence, and the paper restricts its remaining conclusions to curated, template-based prerequisite QA.

Question-Disjoint and Target-Concept-Disjoint Leakage Controls

The paper's **headline claims are intentionally retrieval-only**: they cover retrieval performance on two curated prerequisite DAGs (**LectureBank-Full** and **MOOC-CS**) plus a distinct-domain fact-graph validation on **QASC**. End-to-end **generation outcomes are reported only as diagnostics of context quality**, not as headline results, and an auxiliary denser **HotpotQA** slice is included as an external-validity boundary check rather than a primary benchmark. **Qualitative manual bundles** are released for future audits and are explicitly not used to validate automatic judges or to support inferential claims. The abstract makes this scope explicit by stating that adaptive depth gating is treated as an **optional extension rather than a core claim**, while diffusion with role-aware quotas serves only as a **controlled test case**.

Retrieval-Only Scope of Headline Claims

**Length-aware selection and truncation for token-budgeted RAG** is a strand of prior work that treats the construction of an LLM's input context under a hard length or token budget as an explicit **constrained selection / compression problem**, rather than as fixed-$$k$$ retrieval.

Three representative formulations define this strand:

- **Knapsack-style extractive selection (Riedhammer et al., 2008).** Length-constrained summarization is cast as a knapsack-packing problem: choose a subset of candidate units (e.g., sentences or utterances) that maximizes a utility score subject to a hard length cap. The problem is NP-hard but admits near-optimal heuristic and global-optimization solvers. This is the canonical formulation of *budgeted selection under a length constraint*.
- **Prompt compression for LLM inference (LLMLingua; Jiang et al., 2023).** Given a target token budget, a **budget controller** allocates compression ratios across prompt components, and an **iterative token-level compression** procedure shortens the prompt while preserving the information most relevant to the downstream task. This treats budget satisfaction as a *compression* problem layered on top of selection.
- **List-adaptive retrieval truncation for RAG (Xu et al., 2024).** A joint reranking-and-truncation model **dynamically chooses the cut-off point** of the retrieved list per query, trading off coverage of relevant material against the risk of feeding misinformation or noise into the generator. This treats budget satisfaction as a *learned, query-conditioned truncation* policy on the retrieved list.

The shared commitment across these formulations is that the budget is a **first-class constraint**: the system explicitly decides what to keep, what to drop, and what to compress in order to respect it. A new method in this strand would normally be evaluated by how well its **selection or compression policy** preserves downstream task quality under matched token budgets, against knapsack, compression, and learned-truncation baselines.

Length-Aware Selection and Truncation for Token-Budgeted RAG

The paper's **token-cap analysis** is positioned as a diagnostic tool, not a new compression method. Starting from the observation that **fixed-$$k$$ retrieval does not imply fixed token budgets**, the paper applies a hard token cap on top of an already-selected candidate set and measures how performance moves. Because the underlying selection is held fixed across compared systems, the analysis isolates two specific factors: (i) **ordering** — which retrieved items survive the cap and in what rank, and (ii) **serialization** — how the surviving items are concatenated into the LLM input. The contribution is therefore an audit of how prerequisite-graph-aware retrieval interacts with a length budget under strict parity, rather than a new length-aware compressor in the style of knapsack-based selection or prompt compression.

Learn Before

Related

Learn After