Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/emnlp-auditable-strict-parity-evaluation-of-prerequisite-graph-retrieval-for-rag-under-leakage-controls.pdf
Paper title: Auditable Strict-Parity Evaluation of Prerequisite-Graph Retrieval for RAG under Leakage Controls
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Google

Involves documenting data following a series of steps and interpreting and analyzing the results to increase knowledge.

Research

Involves focusing on a certain area of research to increase understanding in that field. This includes disciplines such as chemistry, economics, psychology, etc.

Disciplinary Research

Legal research is the process of finding answers to legal questions, or checking and verifying the validity of legal precedent, to be cited in a legal brief, or at trial.

Legal Research

Historical research is a qualitative research technique, in which past events are studied, in order to interpret the relevant facts, examine the causes of those events, and assess what relevance and impact those past events might still be having on events in the present day.

Historical Research

Scientific research is performed by examining and applying systematic, constructed scientific methods in order to obtain, study, analyze, and interpret relevant data.

Scientific Research

Add reference nodes for Research as children here.

Research References



Research methods are the techniques and strategies used while collecting or analyzing data.

Research Methods

Research philosophy is a broad term that encompasses the researcher's fundamental beliefs about the nature of knowledge (epistemology), reality (ontology), and values. Research philosophy has an overarching influence on the entire research process, including but not limited to research question framing, research design, methodology, data collection and analysis, and conclusion framing, etc.

Research Philosophy

Research Center

An empirical research report is a formal article that presents the results of one or more new studies. The standard sections of an empirical research report provide a structured outline for presenting the research, detailing the specific methodology, findings, and conclusions.

Empirical Research Report

Root URL: https://doi.org/
Source URL: https://doi.org/10.1609/aaai.v33i01.33016674
Additional cited root URL: https://ojs.aaai.org
Publication title: What Should I Learn First: Introducing LectureBank for NLP Education and Prerequisite Chain Learning
Authors: Irene Li, Alexander R. Fabbri, Robert R. Tung, Dragomir R. Radev
Year: 2019
DOI: 10.1609/aaai.v33i01.33016674
Publisher: AAAI (Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33)
Researcher-agent use: Supports missing prerequisite "LectureBank-Full Prerequisite QA Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: What Should I Learn First: Introducing LectureBank for NLP Education and Prerequisite Chain Learning

Root URL: https://doi.org/
Source URL: https://doi.org/10.18653/v1/2020.coling-main.99
Publication title: R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning
Authors: Irene Li, Vanessa Yan, Tianxiao Li, Rihao Qu, Dragomir Radev
Year: 2020
DOI: 10.18653/v1/2020.coling-main.99
Publisher: COLING 2020 (ACL Anthology)
Researcher-agent use: Supports missing prerequisite "LectureBank-Full Prerequisite QA Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning

Root URL: https://ojs.aaai.org/
Source URL: https://ojs.aaai.org/index.php/AAAI/article/view/4638
Researcher-agent use: Supports missing prerequisite "LectureBank Dataset" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: ojs.aaai.org

Root URL: https://doi.org/
Source URL: https://doi.org/10.18653/v1/P17-1133
Additional cited root URL: https://aclanthology.org
Publication title: Prerequisite Relation Learning for Concepts in MOOCs
Authors: Liangming Pan, Chengjiang Li, Juanzi Li, Jie Tang
Year: 2017
DOI: 10.18653/v1/P17-1133
Publisher: Association for Computational Linguistics (ACL 2017)
Researcher-agent use: Supports missing prerequisite "MOOC-CS Prerequisite Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Prerequisite Relation Learning for Concepts in MOOCs

Root URL: http://moocdata.cn/
Source URL: http://moocdata.cn/data/prerequisite-relation
Publication title: Course Prerequisite Relation (MOOC prerequisite dataset release page)
Authors: Tsinghua KEG
Year: 2017
Publisher: Tsinghua University Knowledge Engineering Group (moocdata.cn)
Researcher-agent use: Supports missing prerequisite "MOOC-CS Prerequisite Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Course Prerequisite Relation (MOOC prerequisite dataset release page)

Root URL: https://doi.org/
Source URL: https://doi.org/10.18653/v1/2020.acl-main.285
Publication title: MOOCCube: A Large-scale Data Repository for NLP Applications in MOOCs
Authors: Jifan Yu, Gan Luo, Tong Xiao, Qingyang Zhong, Yuquan Wang, Wenzheng Feng, Junyi Luo, Chenyu Wang, Lei Hou, Juanzi Li, Zhiyuan Liu, Jie Tang
Year: 2020
DOI: 10.18653/v1/2020.acl-main.285
Publisher: Association for Computational Linguistics (ACL 2020)
Researcher-agent use: Supports missing prerequisite "MOOC-CS Prerequisite Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: MOOCCube: A Large-scale Data Repository for NLP Applications in MOOCs

Root URL: https://doi.org/ Source URL: https://doi.org/10.1609/aaai.v34i05.6319 Publication title: QASC: A Dataset for Question Answering via Sentence Composition Authors: Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, Ashish Sabharwal Year: 2020 DOI: 10.1609/aaai.v34i05.6319 Publisher: AAAI (Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, 34(05), 8082-8090)

Reference: QASC: A Dataset for Question Answering via Sentence Composition

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/1910.11473
Publication title: QASC: A Dataset for Question Answering via Sentence Composition (arXiv preprint)
Authors: Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, Ashish Sabharwal
Year: 2019
Publisher: arXiv:1910.11473
Researcher-agent use: Supports missing prerequisite "QASC Question Answering Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: QASC: A Dataset for Question Answering via Sentence Composition (arXiv preprint)

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.2112.01488
Publication title: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Authors: Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, Matei Zaharia
Year: 2022
DOI: 10.48550/arXiv.2112.01488
Publisher: NAACL 2022 (Association for Computational Linguistics)
Researcher-agent use: Supports missing prerequisite "ColBERTv2 Late-Interaction Retriever" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

Root URL: https://doi.org/
Source URL: https://doi.org/10.1145/3397271.3401075
Publication title: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Authors: Omar Khattab, Matei Zaharia
Year: 2020
DOI: 10.1145/3397271.3401075
Publisher: SIGIR 2020 (ACM)
Researcher-agent use: Supports missing prerequisite "ColBERTv2 Late-Interaction Retriever" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2004.12832
Researcher-agent use: Supports missing prerequisite "Late-Interaction Neural Retrieval" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: arxiv.org

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2112.01488
Researcher-agent use: Supports missing prerequisite "ColBERTv2 Late-Interaction Retriever" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Root URL: https://nlp.stanford.edu/ Source URL: https://nlp.stanford.edu/IR-book/information-retrieval-book.html Publication title: Introduction to Information Retrieval Authors: Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze Year: 2008 Publisher: Cambridge University Press

Reference: Introduction to Information Retrieval

Root URL: https://en.wikipedia.org/
Source URL: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)
Publication title: Evaluation measures (information retrieval)
Authors: Wikipedia contributors
Year: 2024
Publisher: Wikipedia
Researcher-agent use: Supports missing prerequisite "Recall@k Retrieval Metric" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Evaluation measures (information retrieval)

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.2301.12652
Publication title: REPLUG: Retrieval-Augmented Black-Box Language Models
Authors: Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
Year: 2023
DOI: 10.48550/arXiv.2301.12652
Publisher: arXiv (cs.CL); later appeared at NAACL 2024
Researcher-agent use: Supports missing prerequisite "RePlug Retrieval-Augmented Black-Box Language Model" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: REPLUG: Retrieval-Augmented Black-Box Language Models

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2301.12652
Researcher-agent use: Supports missing prerequisite "RePlug Retrieval-Augmented Black-Box Language Model" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Root URL: https://doi.org/
Source URL: https://doi.org/10.18653/v1/D18-1259
Publication title: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Authors: Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning
Year: 2018
DOI: 10.18653/v1/D18-1259
Publisher: Association for Computational Linguistics (EMNLP 2018)
Researcher-agent use: Supports missing prerequisite "HotpotQA Multi-Hop QA Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.1809.09600
Publication title: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (arXiv preprint)
Authors: Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning
Year: 2018
DOI: 10.48550/arXiv.1809.09600
Publisher: arXiv
Researcher-agent use: Supports missing prerequisite "HotpotQA Multi-Hop QA Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (arXiv preprint)

Root URL: https://hotpotqa.github.io/
Source URL: https://hotpotqa.github.io/
Publication title: HotpotQA Official Dataset and Leaderboard
Authors: Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning
Year: 2018
Publisher: Official project website
Researcher-agent use: Supports missing prerequisite "HotpotQA Multi-Hop QA Benchmark" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: HotpotQA Official Dataset and Leaderboard

Root URL: https://doi.org/
Source URL: https://doi.org/10.18653/v1/2020.emnlp-main.550
Publication title: Dense Passage Retrieval for Open-Domain Question Answering
Authors: Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih
Year: 2020
DOI: 10.18653/v1/2020.emnlp-main.550
Publisher: Association for Computational Linguistics (EMNLP 2020)
Researcher-agent use: Supports missing prerequisite "Dense Passage Retrieval (DPR)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Dense Passage Retrieval for Open-Domain Question Answering

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.2004.04906
Publication title: Dense Passage Retrieval for Open-Domain Question Answering (arXiv preprint)
Authors: Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih
Year: 2020
DOI: 10.48550/arXiv.2004.04906
Publisher: arXiv
Researcher-agent use: Supports missing prerequisite "Dense Passage Retrieval (DPR)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Dense Passage Retrieval for Open-Domain Question Answering (arXiv preprint)

**LectureBank** is a publicly released corpus of English lecture slide files collected from university courses in NLP, ML, AI, deep learning, and information retrieval. The original release contains $$1{,}352$$ lecture files and $$208$$ manually annotated prerequisite-relation topic pairs. Each topic is classified against a fixed taxonomy, and each ordered pair of topics is labeled as a prerequisite link or not. The dataset is the canonical resource for prerequisite-chain learning in NLP education.

LectureBank Dataset

**MOOC-CS** is a computer-science-focused prerequisite-relation benchmark assembled from Massive Open Online Course (MOOC) materials, originating from Pan et al. (ACL 2017). It comprises Coursera lecture captions in the **Data Structures and Algorithms** and **Machine Learning** domains, with course core concepts extracted from the captions and manually annotated ordered concept pairs labeled as prerequisite or not. Each course release provides three file types: a JSON Captions file (tokenized, POS-tagged video transcripts), a Core_concepts file listing per-domain concepts, and a Labeled file storing concept pairs with their prerequisite labels. The benchmark is used to evaluate prerequisite-link retrieval systems, with a fixed candidate concept pool and ranking metrics such as Recall@$$k$$. The default release is English-only, which is why graph-aware retrieval studies report separately on language-matched query and encoder controls when extending evaluations to multilingual MOOC settings.

MOOC-CS Prerequisite Benchmark

**QASC** (Question Answering via Sentence Composition) is a multi-hop multiple-choice science QA benchmark. It contains $$9{,}980$$ grade-school-level science questions, each with $$8$$ answer choices, together with an associated corpus of roughly $$17$$ million science sentences. Every question is annotated with two corpus facts that must be retrieved and composed to support the correct answer, and the decomposition into those facts is not evident from the question surface form, so retrieval must introduce new concepts or relations. The standard task is to choose the correct answer given the question and choices, and retrieval systems are evaluated by their ability to recover the two annotated supporting sentences from the corpus, typically with Recall@$$k$$ over a fixed candidate pool. Headline comparisons fix the encoder, candidate pool, cutoff, matching rule, and split policy so that retrievers (e.g., dense retrievers, ColBERTv2, RePlug) can be compared under matched conditions.

QASC Question Answering Benchmark

**Late-interaction neural retrieval** is a paradigm in neural information retrieval where queries and documents are independently encoded into **multi-vector representations**, with one contextualized embedding per token, and relevance is computed at query time through fine-grained token-level interactions rather than a single dot product between pooled embeddings. The standard scoring function is **MaxSim**: for each query token embedding, the maximum similarity across all document token embeddings is taken, and the per-query-token maxima are summed to produce the query-document score. This decouples encoding from interaction (so document embeddings can be precomputed and indexed) while preserving expressive fine-grained matching, at the cost of substantially larger indexes than single-vector dense retrievers.

Late-Interaction Neural Retrieval

**Recall@$$k$$** (written **R@$$k$$**) is a top-$$k$$ ranking evaluation metric used in information retrieval. For a single query with a set $$R$$ of gold relevant items and a ranked list of retrieved items, let $$R_k$$ be the set of items appearing in the top $$k$$ positions. Then

$$\mathrm{Recall@}k = \frac{|R \cap R_k|}{|R|}.$$

The corpus-level score averages this quantity over a fixed evaluation set of queries. Recall@$$k$$ takes values in $$[0,1]$$ and is **rank-agnostic within the top-$$k$$ window**: it only counts how many relevant items fall in the top $$k$$, not their internal ordering. The cutoff $$k$$ must be reported alongside the score (e.g. R@$$10$$), and meaningful headline comparisons fix the encoder, candidate pool, cutoff, matching rule, and split policy so that two systems compute the same numerator and denominator.

Recall@k Retrieval Metric

**RePlug** is a retrieval-augmented language modeling framework introduced by Shi et al. (2023) that treats the language model as a frozen **black box**: the LM's parameters and architecture are never modified, and the only interface used is input-output. Given a query, a dense retriever fetches the top-$$k$$ documents from an external corpus, each retrieved document is independently prepended to the query, and the resulting per-document LM next-token distributions are combined via a retrieval-score-weighted ensemble. An optional extension, **RePlug LSR** (LM-Supervised Retrieval), tunes the retriever using the language model's own likelihood of the target output as a supervision signal, so the retriever learns to surface documents that actually improve LM predictions. Because the LM is untouched, RePlug can be paired with any off-the-shelf retriever (e.g. a late-interaction retriever such as ColBERTv2) and any black-box LM, which is why the combined **ColBERTv2/RePlug** system can appear as a strict-parity baseline in retrieval-augmented QA evaluations.

RePlug Retrieval-Augmented Black-Box Language Model

**HotpotQA** is a large-scale multi-hop question answering dataset introduced by Yang et al. at EMNLP 2018. It contains approximately $$113{,}000$$ English question-answer pairs built over Wikipedia, where each question is constructed to require reasoning over **two or more supporting paragraphs** rather than a single passage. Every question is annotated at the sentence level with **supporting facts** that justify the answer, enabling joint evaluation of answer correctness and evidence selection. The benchmark defines two standard settings: a **distractor** setting, in which a system must answer each question given $$10$$ paragraphs ($$2$$ gold plus $$8$$ lexically related distractors), and a **fullwiki** open-domain setting, in which the supporting paragraphs must be retrieved from the entire Wikipedia corpus. The release also includes **comparison** questions that contrast attributes of two entities. Standard metrics are Exact Match and F1 for both the predicted answer and the predicted supporting-fact set, with retrieval components typically scored by Recall@$$k$$ over the fullwiki candidate pool.

HotpotQA Multi-Hop QA Benchmark

**Single-vector dense passage retrieval** is a neural information-retrieval paradigm in which a query and each candidate passage are independently mapped to a **single fixed-dimensional dense embedding** by neural encoders, and relevance is scored as a similarity (typically the dot product or cosine similarity) between those two pooled vectors. Because each passage is represented by exactly one vector, the corpus can be pre-encoded once and indexed with a maximum-inner-product search (MIPS) structure such as FAISS, so retrieval at query time reduces to a nearest-neighbor lookup against the query embedding. This contrasts with multi-vector / late-interaction retrievers, which keep one contextualized embedding per token and defer fine-grained matching to query time at the cost of much larger indexes. Single-vector dense retrievers are the standard 'flat dense retrieval' baseline in modern retrieval-augmented generation pipelines.

Single-Vector Dense Passage Retrieval

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2005.11401
Publication title: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
Year: 2020
Publisher: Advances in Neural Information Processing Systems 33 (NeurIPS 2020)
Researcher repair helper created this source-specific Reference after detecting host-only Reference reuse.

Reference: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.2506.06331
Publication title: How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
Authors: Qiming Zeng, Xiao Yan, Hao Luo, Yuhao Lin, Yuxiang Wang, Fangcheng Fu, Bo Du, Quanqing Xu, Jiawei Jiang
Year: 2025
DOI: 10.48550/arXiv.2506.06331
Publisher: arXiv
Researcher-agent use: Supports missing prerequisite "Unbiased Graph-RAG Evaluation Protocol (Zeng et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2506.06331
Researcher-agent use: Supports missing prerequisite "Unbiased GraphRAG Evaluation Framework (Zeng et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

**Unbiased GraphRAG evaluation framework** is a methodological proposal by Zeng et al. (2025) targeting answer-evaluation pipelines for Graph Retrieval-Augmented Generation (GraphRAG). The authors identify two critical flaws in canonical GraphRAG answer evaluation: (i) **unrelated questions**, where the evaluation questions are not actually grounded in the underlying graph or text corpus, and (ii) **evaluation biases**, where LLM-as-judge assessments systematically favor certain answer styles independent of correctness. The framework's corrective has two parts: **graph-text-grounded question generation**, which constructs evaluation questions from anchor subgraphs together with their supporting text so the questions are answerable from the corpus, and an **unbiased judging procedure** that mitigates LLM-judge biases. Applied to three representative GraphRAG systems, the framework reports that previously claimed performance gains are substantially more moderate than originally stated, supporting the broader observation that graph-aware RAG comparisons are highly sensitive to evaluation-protocol choices.

Unbiased GraphRAG Evaluation Framework (Zeng et al., 2025)

Root URL: https://doi.org/
Source URL: https://doi.org/10.48550/arXiv.2502.11371
Publication title: RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
Authors: Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiliang Tang, Zhigang Hua, Bo Long, Hui Liu, Charu C. Aggarwal, et al.
Year: 2025
DOI: 10.48550/arXiv.2502.11371
Publisher: arXiv
Researcher-agent use: Supports missing prerequisite "RAG vs Graph-RAG Controlled Comparison (Han et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: RAG vs. GraphRAG: A Systematic Evaluation and Key Insights

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2502.11371
Researcher-agent use: Supports missing prerequisite "RAG vs Graph-RAG Controlled Comparison (Han et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

**RAG vs GraphRAG: A Systematic Evaluation and Key Insights** is a 2025 study by Han et al. that benchmarks **Retrieval-Augmented Generation (RAG)** against **Graph Retrieval-Augmented Generation (GraphRAG)** under a **unified evaluation protocol**. The protocol fixes data preprocessing, retrieval configurations, and generation settings across compared systems so that observed differences can be attributed to the retrieval paradigm rather than to interface mismatches. The benchmark covers question answering and query-based summarization on established text corpora, and the analysis additionally examines failure modes, efficiency trade-offs, and LLM-judge evaluation biases. The authors show that RAG and GraphRAG have distinct task-dependent strengths, and that hybrid selection and integration strategies can combine those strengths for more consistent performance. The work is cited as motivating evidence that fair graph-vs-flat comparisons require matched protocols.

RAG vs Graph-RAG Controlled Comparison (Han et al., 2025)

Root URL: https://arxiv.org/
Source URL: https://arxiv.org/abs/2506.20051
Publication title: Controlled Retrieval-augmented Context Evaluation for Long-form RAG
Authors: Jia-Huei Ju, Suzan Verberne, Maarten de Rijke, Andrew Yates
Year: 2025
Publisher: arXiv preprint; Findings of the Association for Computational Linguistics: EMNLP 2025
Researcher-agent use: Supports missing prerequisite "CRUX Controlled RAG Context Evaluation (Ju et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Controlled Retrieval-augmented Context Evaluation for Long-form RAG

Root URL: https://aclanthology.org/
Source URL: https://aclanthology.org/2025.findings-emnlp.1151/
Publication title: Controlled Retrieval-augmented Context Evaluation for Long-form RAG (ACL Anthology)
Authors: Jia-Huei Ju, Suzan Verberne, Maarten de Rijke, Andrew Yates
Year: 2025
Publisher: Association for Computational Linguistics (Findings of EMNLP 2025)
Researcher-agent use: Supports missing prerequisite "CRUX Controlled RAG Context Evaluation (Ju et al., 2025)" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Controlled Retrieval-augmented Context Evaluation for Long-form RAG (ACL Anthology)

**CRUX** (**C**ontrolled **R**etrieval-**a**ugmented Conte**x**t Evaluation) is a RAG evaluation framework introduced by Ju et al. (2025) that directly assesses the **retrieval-augmented context** delivered to the generator, rather than scoring the retrieval module with standard relevance ranking metrics alone. The framework uses **human-written summaries** to fix the information scope of each query and a **question-based scoring** procedure: questions derived from the controlled summary are answered against the retrieved context, so the score measures how well the context covers the information actually needed for long-form generation. Because the information scope is held constant across compared systems, CRUX provides a controlled, diagnostic view of retrieval quality and is positioned as a more reflective alternative to relevance-only retrieval metrics for long-form RAG.

CRUX Controlled RAG Context Evaluation (Ju et al., 2025)

Root URL: https://doi.org/
Source URL: https://doi.org/10.1613/jair.2096
Publication title: Anytime Heuristic Search
Authors: Eric A. Hansen, Rong Zhou
Year: 2007
DOI: 10.1613/jair.2096
Publisher: Journal of Artificial Intelligence Research
Researcher-agent use: Supports missing prerequisite "Anytime Search Under Resource Constraints" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Reference: Anytime Heuristic Search

Root URL: https://doi.org/
Source URL: https://doi.org/10.1016/S0169-7552(98)00110-X
Publication title: The Anatomy of a Large-Scale Hypertextual Web Search Engine
Authors: Sergey Brin, Lawrence Page
Year: 1998
DOI: 10.1016/S0169-7552(98)00110-X
Publisher: Computer Networks and ISDN Systems (Elsevier) / WWW7
Researcher-agent use: Supports missing prerequisite "Personalized / Local PageRank Propagation" for the Researcher agent.

This Reference node represents a publication or authoritative source used by the Researcher agent to create prerequisite knowledge before citing paper-derived nodes.

Learn Before

Related