The TF-IDF version of the document scoring formula computes the cosine similarity between the TF-IDF vector of a query $$q$$ and the TF-IDF vector of a document $$d$$:

$$score(q,d) = \sum_{t \in q} \frac{tf-idf(t,q)}{\sqrt{\sum_{q_i \in q} tf-idf^2(q_i,q)}} \cdot \frac{tf-idf(t,d)}{\sqrt{\sum_{d_i \in d} tf-idf^2(d_i,d)}}$$

Where:
- $$t$$ is a term present in the query $$q$$ and the document $$d$$.
- $$tf-idf(t,q)$$ is the TF-IDF weight of term $$t$$ in query $$q$$.
- $$tf-idf(t,d)$$ is the TF-IDF weight of term $$t$$ in document $$d$$.
- The denominators represent the Euclidean ($$L_2$$) norms of the query and document vectors, respectively, which normalize the score to be independent of document and query lengths.

TF-IDF Document Scoring Formula

The BM25 formula is a document scoring method in information retrieval that improves upon TF-IDF. It introduces parameters to limit the upper boundary of the term frequency (TF) value and uses average document length to measure influence. The score for a query $$q$$ and document $$d$$ is defined as:

$$\sum_{t \in q} \frac{N}{df_t} \cdot \frac{tf_{t,d}}{k(1-b+ b \frac{|d|}{|d_{avg}|}) + tf_{t,d}}$$

BM25 Document Scoring Formula

Document is scored according to the similarity with query to decide the rank of it in the response list. The plain score can be expressed as:
$score(q,d) = cos(q,d) = \frac{\textbf(q)}{|\textbf(q)|} \cdot \frac{\textbf(d)}{|\textbf(d)|} $

University of Michigan - Ann Arbor

It is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. 

Information Retrieval

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms: - Term frequency is the frequency of the word $$t$$ in the document $$d$$, commonly denoted in log weighting as $$tf_{t, d} = log_{10}(count(t, d) + 1)$$. - Inverse document frequency is also usually denoted in a log function as $$idf_t = log_{10}\left(\frac{N}{df_t}\right)$$, where document frequency $$df_t$$ of a term $$t$$ is the number of documents in which it occurs and $$N$$ is the total number of documents in the collection. The tf-idf weighted value $$w_{t, d}$$ for word $$t$$ in document $$d$$ is thus defined as $$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The system relies on the massive information in the collected data resource. Given a user question, information retrieval is used to find relevant
passages. Then neural reading comprehension algorithms read these retrieved passages and draw an answer directly from spans of text.

Information-retrieval (IR) based QA

A document is whatever unit of text the system indexes and retrieves (web pages, scientific papers, news articles, or even shorter passages like paragraphs.

Document(In IR)

 A collection refers to a set of documents being used to satisfy user requests. 

Collection(In IR)

 A term refers to a word in a collection, but it may also include phrases.

Term(In IR)

A query represents a user’s information need expressed as a set of terms.

Query(In IR)

Term Frequency(TF)

Document Scoring

In the past it was common to remove high-frequency words from both the query and document before representing them. The list of such high-frequency words to be removed is called a stop list. 

Stop Words

It is a database index storing a mapping from content, such as words or numbers, to its locations in a table, or in a document or a set of documents (named in contrast to a forward index, which maps from documents to content). 

Inverted Index

Similar to traditional information retrieval (IR) systems, a datastore can be utilized to manage and retrieve relevant texts. This involves storing a collection of documents, often with full-text indexing like in a search engine, to enable searching for and accessing documents that match a given text-based query.

Using Datastores for Text Retrieval

Retrieval-Augmented Generation (RAG) is a context scaling approach that dynamically incorporates timely or specialized external knowledge into a large language model's input. RAG systems first retrieve relevant document snippets from a large collection or database based on the current query. These retrieved pieces of information are then added to the prompt context, grounding the model's predictions in specific external sources to generate responses that are both relevant to the input and factually up-to-date.

Retrieval-Augmented Generation (RAG)

The tf-idf model can be used to compute word similarity by computing the cosine of two word vectors

The tf-idf vector models can also be used to decide if two documents are similar. 
For each document that we want to compare,  we need to compute the centroid for all their vectors. 
Given $$k$$ word vectors $$w_1,w_2,....,w_k$$, the centroid document vector $$d$$ is:

$$
d = \frac{{w_1+w_2+...+w_k}}{k}
$$

Then we can compute the $$cos(d_1,d_2)$$ to estimate the similarity of these two documents: high cosine, high similarity.

Application of the tf-idf vector models

 $idf_t = log_{10} \frac{N}{df_t}$ where N is the total number of documents in the collection.

Learn Before

Related

Learn After