In the past it was common to remove high-frequency words from both the query and document before representing them. The list of such high-frequency words to be removed is called a stop list. 

University of Michigan - Ann Arbor

It is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. 

Information Retrieval

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The system relies on the massive information in the collected data resource. Given a user question, information retrieval is used to find relevant
passages. Then neural reading comprehension algorithms read these retrieved passages and draw an answer directly from spans of text.

Information-retrieval (IR) based QA

A document is whatever unit of text the system indexes and retrieves (web pages, scientific papers, news articles, or even shorter passages like paragraphs.

Document(In IR)

 A collection refers to a set of documents being used to satisfy user requests. 

Collection(In IR)

 A term refers to a word in a collection, but it may also include phrases.

Term(In IR)

A query represents a user’s information need expressed as a set of terms.

Query(In IR)

Term Frequency(TF)

The document frequency $df_t$ of a term t is the number of documents it occurs in.

Document Frequency(DF)

Document is scored according to the similarity with query to decide the rank of it in the response list. The plain score can be expressed as:
$score(q,d) = cos(q,d) = \frac{\textbf(q)}{|\textbf(q)|} \cdot \frac{\textbf(d)}{|\textbf(d)|} $

Document Scoring

$\sum_{t \in q} \frac{N}{df_t} \cdot \frac{tf_{t,d}}{k(1-b+ b \frac{|d|}{|d_{avg}|}) + tf_{t,d}}$

BM25

Stop Words

It is a database index storing a mapping from content, such as words or numbers, to its locations in a table, or in a document or a set of documents (named in contrast to a forward index, which maps from documents to content). 

Inverted Index

Similar to traditional information retrieval (IR) systems, a datastore can be utilized to manage and retrieve relevant texts. This involves storing a collection of documents, often with full-text indexing like in a search engine, to enable searching for and accessing documents that match a given text-based query.

Using Datastores for Text Retrieval

Retrieval-Augmented Generation (RAG) is a context scaling approach that dynamically incorporates timely or specialized external knowledge into a large language model's input. RAG systems first retrieve relevant document snippets from a large collection or database based on the current query. These retrieved pieces of information are then added to the prompt context, grounding the model's predictions in specific external sources to generate responses that are both relevant to the input and factually up-to-date.

Learn Before

Related