Perplexity is a probability-based metric for evaluating language models. It is the weighted average of the number of possible next words that can follow any word, a.k.a. the weighted average branching factor.  

Given a mini-language of 10 words "zero, one ... ten", each word's occurrence probability is 1/10 (unigram), the perplexity is the inverse is 10:

$$
\begin{aligned}
\mathrm{PP}(W) &=P\left(w_{1} w_{2} \ldots w_{N}\right)^{-\frac{1}{N}} \\
&=\left(\frac{1}{10}\right)^{-\frac{1}{N}} \\
&=\frac{1}{10} \\
&=10
\end{aligned}
$$

University of Michigan - Ann Arbor

An n-gram is a sequence of n tokens, where training models based on this are straightforward and have been used as the building block of statistical language modeling for many years. There are also specific names for small values of n, such as unigram for n - 1, or bigram for n - 2, and trigram for n - 3.

n-grams

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The assumption that the probability of a word depends only on the previous word is called a Markov assumption.

Markov used in NLP

Perplexity

The general equations of an n-gram model apply the Markov assumption to estimate word probabilities. The conditional probability for the next word is approximated by looking $$N-1$$ words into the past:

$$P(w_n|w_{1:n-1}) \approx P(w_n|w_{n-N+1:n-1})$$

The probability of a complete word sequence is approximated as the product of these conditional probabilities:

$$P(w_{1:n}) \approx \prod^n_{k=1}P(w_k|w_{k-N+1:k-1})$$

Learn Before

Related