The TF-IDF version of the document scoring formula computes the cosine similarity between the TF-IDF vector of a query $$q$$ and the TF-IDF vector of a document $$d$$:

$$score(q,d) = \sum_{t \in q} \frac{tf-idf(t,q)}{\sqrt{\sum_{q_i \in q} tf-idf^2(q_i,q)}} \cdot \frac{tf-idf(t,d)}{\sqrt{\sum_{d_i \in d} tf-idf^2(d_i,d)}}$$

Where:
- $$t$$ is a term present in the query $$q$$ and the document $$d$$.
- $$tf-idf(t,q)$$ is the TF-IDF weight of term $$t$$ in query $$q$$.
- $$tf-idf(t,d)$$ is the TF-IDF weight of term $$t$$ in document $$d$$.
- The denominators represent the Euclidean ($$L_2$$) norms of the query and document vectors, respectively, which normalize the score to be independent of document and query lengths.

TF-IDF Document Scoring Formula

The BM25 formula is a document scoring method in information retrieval that improves upon TF-IDF. It introduces parameters to limit the upper boundary of the term frequency (TF) value and uses average document length to measure influence. The score for a query $$q$$ and document $$d$$ is defined as:

$$\sum_{t \in q} \frac{N}{df_t} \cdot \frac{tf_{t,d}}{k(1-b+ b \frac{|d|}{|d_{avg}|}) + tf_{t,d}}$$

BM25 Document Scoring Formula

A document is scored according to its similarity with a query to determine its rank in a retrieved response list. The fundamental score can be expressed using cosine similarity between the query vector $$\mathbf{q}$$ and the document vector $$\mathbf{d}$$: $$score(q,d) = \cos(q,d) = \frac{\mathbf{q}}{|\mathbf{q}|} \cdot \frac{\mathbf{d}}{|\mathbf{d}|}$$

University of Michigan - Ann Arbor

Google

It is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. 

Information Retrieval

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The system relies on the massive information in the collected data resource. Given a user question, information retrieval is used to find relevant
passages. Then neural reading comprehension algorithms read these retrieved passages and draw an answer directly from spans of text.

Information-retrieval (IR) based QA

A document is whatever unit of text the system indexes and retrieves (web pages, scientific papers, news articles, or even shorter passages like paragraphs.

Document(In IR)

 A collection refers to a set of documents being used to satisfy user requests. 

Collection(In IR)

 A term refers to a word in a collection, but it may also include phrases.

Term(In IR)

A query represents a user’s information need expressed as a set of terms.

Query(In IR)

Term Frequency(TF)

In the past it was common to remove high-frequency words from both the query and document before representing them. The list of such high-frequency words to be removed is called a stop list. 

Stop Words

It is a database index storing a mapping from content, such as words or numbers, to its locations in a table, or in a document or a set of documents (named in contrast to a forward index, which maps from documents to content). 

Inverted Index

Similar to traditional information retrieval (IR) systems, a datastore can be utilized to manage and retrieve relevant texts. This involves storing a collection of documents, often with full-text indexing like in a search engine, to enable searching for and accessing documents that match a given text-based query.

Using Datastores for Text Retrieval

Retrieval-Augmented Generation (RAG) is a context scaling approach that dynamically incorporates timely or specialized external knowledge into a large language model's input. RAG systems first retrieve relevant document snippets from a large collection or database based on the current query. These retrieved pieces of information are then added to the prompt context, grounding the model's predictions in specific external sources to generate responses that are both relevant to the input and factually up-to-date.

Learn Before

Related

Learn After