The probability for a Katz backoff n-gram is given by $$P_{BO}(w_n|w_{n-N+1:n-1}) = \begin{cases} P^*(w_n|w_{n-N+1:n-1}), \text{ if } C(w_{n-N+1:n}) > 0 \\\\ \alpha(w_{n-N+1:n-1})P_{BO}(w_n|w_{n-N+2:n-1}), \text{ otherwise} \end{cases}$$ where $$\alpha$$ is a function that distributes higher-order n-grams to lower-order n-grams, to ensure that the backoff model gives a correct probability distribution.

Katz Backoff Equation

Katz Backoff is a kind of backoff where we rely on a discounted probability P* if we have seen this n-gram before. Otherwise, we recursively back off to the Katz probability for the (n-1)-gram.

University of Michigan - Ann Arbor

Backoff is the method of estimating probabilities using to a lower-order n-gram if we have zero evidence for a higher-order n-gram; if the n-gram we need has zero counts, we approximate it by backing off to the (n-1)-gram, and continue backing off until we reach a history that has some counts.

Backoff

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Stupid backoff gives up the idea of trying to make the language model a true probability distribution, and also does not discount the higher-order probabilities. If a higher-order n-gram has a zero count, we simply backoff to a lower order n-gram, weighed by a fixed (context-independent) weight $$\lambda$$, for which a value of 0.4 is found to work well. The stupid backoff applied to n-gram is given by: $$S(w_i | w^{i-1}_{i-k+1}) = \begin{cases} \frac{\text{count}(w^i_{i-k+1})}{\text{count}(w^{i-1}_{i-k+1})} &\text{ if count}(w^i_{i-k+1}) > 0 \\ \lambda S(w_i|w^{i-1}_{i-k+2}) &\text{ otherwise} \end{cases}$$ The backoff terminates in the unigram, which has probability $$S(w) = \frac{\text{count}(w)}{N}$$

Learn Before

Related

Learn After

Learn After

Related