 $idf_t = log_{10} \frac{N}{df_t}$ where N is the total number of documents in the collection.

University of Michigan - Ann Arbor

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms: - Term frequency is the frequency of the word $$t$$ in the document $$d$$, commonly denoted in log weighting as $$tf_{t, d} = log_{10}(count(t, d) + 1)$$. - Inverse document frequency is also usually denoted in a log function as $$idf_t = log_{10}\left(\frac{N}{df_t}\right)$$, where document frequency $$df_t$$ of a term $$t$$ is the number of documents in which it occurs and $$N$$ is the total number of documents in the collection. The tf-idf weighted value $$w_{t, d}$$ for word $$t$$ in document $$d$$ is thus defined as $$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

The document frequency $$df_t$$ of a term $$t$$ is the number of documents in a collection that contain the term.

Document Frequency (DF)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The tf-idf model can be used to compute word similarity by computing the cosine of two word vectors

The tf-idf vector models can also be used to decide if two documents are similar. 
For each document that we want to compare,  we need to compute the centroid for all their vectors. 
Given $$k$$ word vectors $$w_1,w_2,....,w_k$$, the centroid document vector $$d$$ is:

$$
d = \frac{{w_1+w_2+...+w_k}}{k}
$$

Then we can compute the $$cos(d_1,d_2)$$ to estimate the similarity of these two documents: high cosine, high similarity.

Application of the tf-idf vector models

Inverse Document Frequency(IDF)

Term Frequency(TF)

The BM25 formula is a document scoring method in information retrieval that improves upon TF-IDF. It introduces parameters to limit the upper boundary of the term frequency (TF) value and uses average document length to measure influence. The score for a query $$q$$ and document $$d$$ is defined as:

$$\sum_{t \in q} \frac{N}{df_t} \cdot \frac{tf_{t,d}}{k(1-b+ b \frac{|d|}{|d_{avg}|}) + tf_{t,d}}$$

Learn Before

Related