The power law relationship of Zipf's law can be expressed in an equivalent logarithmic form, which is useful for visualizing the distribution on a log-log plot. This form is written as $$\log n_i = -\alpha \log i + c$$, where $$n_i$$ is the frequency of the $$i^{	extrm{th}}$$ most frequent word, $$\alpha$$ is the distribution's exponent, and $$c$$ is a constant. On a log-log plot, the frequencies of words, excluding a few highly frequent exceptions, roughly form a straight line.

Logarithmic Form of Zipf's Law

The power law distribution described by Zipf's law applies not only to individual words (unigrams) but also to sequences of words, such as bigrams and trigrams, though typically with a smaller exponent $$\alpha$$. Because many $$n$$-grams occur very rarely in a corpus, methods relying solely on counting statistics face a significant sparsity problem and tend to overestimate the frequency of the infrequent combinations. This high frequency of rare occurrences makes counting-based methods inadequate for language modeling and strongly motivates the transition to deep learning models.

Zipf's Law for n-grams and Sparsity

Zipf's law describes the empirical phenomenon where the frequency of words in a corpus decays quickly, following a power law distribution. Specifically, the frequency $$n_i$$ of the $$i^{	extrm{th}}$$ most frequent word is inversely proportional to its rank, mathematically expressed as $$n_i \propto \frac{1}{i^\alpha}$$, where $$\alpha$$ is the exponent characterizing the distribution.

Claude

How often (the frequency) that a word appears in some text or speech.

Word Frequency

Dive into Deep Learning

A collection of utterances and written text that is compiled and estimates the frequency of which words are used (word frequency), the different meanings a words has (lexical ambiguity), and the grammatical constructions used.

Corpus 

We respond faster to words with high frequency in a language than low frequency (we will respond faster to home than hike).

Learn Before

Related

Learn After