The tf-idf model can be used to compute word similarity by computing the cosine of two word vectors

The tf-idf vector models can also be used to decide if two documents are similar. 
For each document that we want to compare,  we need to compute the centroid for all their vectors. 
Given $$k$$ word vectors $$w_1,w_2,....,w_k$$, the centroid document vector $$d$$ is:

$$
d = \frac{{w_1+w_2+...+w_k}}{k}
$$

Then we can compute the $$cos(d_1,d_2)$$ to estimate the similarity of these two documents: high cosine, high similarity.

University of Colorado at Boulder

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms:
- Term frequency is the frequency of the word $t$ in the document $d$, commonly denoted in log weighting as $tf_{t, d} = log_{10}(count(t, d) + 1)$.
-  Inverse document frequency is also usually denoted in a log function as $idf_t = log_{10}\left(\frac{N}{df_t}\right)$, where document frequency $df_t$ of a term $t$ is the number of documents in which it occurs and $N$ is the total number of documents in the collection.
The tf-idf weighted value $w_{t, d}$ for word $t$ in document $d$ is thus defined as 
$$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Application of the tf-idf vector models

Document is scored according to the similarity with query to decide the rank of it in the response list. The plain score can be expressed as:
$score(q,d) = cos(q,d) = \frac{\textbf(q)}{|\textbf(q)|} \cdot \frac{\textbf(d)}{|\textbf(d)|} $

Document Scoring

 $idf_t = log_{10} \frac{N}{df_t}$ where N is the total number of documents in the collection.

Inverse Document Frequency(IDF)

Term Frequency(TF)

$\sum_{t \in q} \frac{N}{df_t} \cdot \frac{tf_{t,d}}{k(1-b+ b \frac{|d|}{|d_{avg}|}) + tf_{t,d}}$

Learn Before

Related