To simplify query processing, an approximate version of the TF-IDF document score can be calculated by normalizing the sum of the TF-IDF weights for matching terms by the document length $$|d|$$ instead of computing the full Euclidean norms of the vectors: $$score(q,d) = \sum_{t \in q} \frac{tf-idf(t,d)}{|d|}$$

University of Michigan - Ann Arbor

Google

The TF-IDF version of the document scoring formula computes the cosine similarity between the TF-IDF vector of a query $$q$$ and the TF-IDF vector of a document $$d$$:

$$score(q,d) = \sum_{t \in q} \frac{tf-idf(t,q)}{\sqrt{\sum_{q_i \in q} tf-idf^2(q_i,q)}} \cdot \frac{tf-idf(t,d)}{\sqrt{\sum_{d_i \in d} tf-idf^2(d_i,d)}}$$

Where:
- $$t$$ is a term present in the query $$q$$ and the document $$d$$.
- $$tf-idf(t,q)$$ is the TF-IDF weight of term $$t$$ in query $$q$$.
- $$tf-idf(t,d)$$ is the TF-IDF weight of term $$t$$ in document $$d$$.
- The denominators represent the Euclidean ($$L_2$$) norms of the query and document vectors, respectively, which normalize the score to be independent of document and query lengths.

TF-IDF Document Scoring Formula

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Learn Before

Related