The general equations of an n-gram model apply the Markov assumption to estimate word probabilities. The conditional probability for the next word is approximated by looking $$N-1$$ words into the past:

$$P(w_n|w_{1:n-1}) \approx P(w_n|w_{n-N+1:n-1})$$

The probability of a complete word sequence is approximated as the product of these conditional probabilities:

$$P(w_{1:n}) \approx \prod^n_{k=1}P(w_k|w_{k-N+1:k-1})$$

General Equations of an N-Gram Model

The assumption that the probability of a word depends only on the previous word is called a Markov assumption.

University of Michigan - Ann Arbor

An n-gram is a sequence of n tokens, where training models based on this are straightforward and have been used as the building block of statistical language modeling for many years. There are also specific names for small values of n, such as unigram for n - 1, or bigram for n - 2, and trigram for n - 3.

n-grams

They are a type of stochastic models that describe a sequence of events. The probability of each event only depends on the state of previous event.

Markov Chains

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Perplexity is a probability-based metric for evaluating language models. It is the weighted average of the number of possible next words that can follow any word, a.k.a. the weighted average branching factor.  

Given a mini-language of 10 words "zero, one ... ten", each word's occurrence probability is 1/10 (unigram), the perplexity is the inverse is 10:

$$
\begin{aligned}
\mathrm{PP}(W) &=P\left(w_{1} w_{2} \ldots w_{N}\right)^{-\frac{1}{N}} \\
&=\left(\frac{1}{10}\right)^{-\frac{1}{N}} \\
&=\frac{1}{10} \\
&=10
\end{aligned}
$$

Perplexity

The authors tested how NTMs would perform as re-writable tables, similar to N-Gram models. They used all possible 6-Gram distributions over binary sequences: the network first observes the data one bit at a time and then must predict the next bit, requiring it to infer and store the underlying N-Gram statistics in memory.

Learn Before

Related

Learn After

Learn After

Related