Kneser-Ney discounting augments absolute discounting to better handle low-order unigram distributions. Rather than estimating the probability of a word w simply appearing, we create a unigram model that estimates the probability of w appearing as a novel continuation, completing some previously unseen bigram. This estimate is based on the number of bigrams w has previously completed, with the intuition that words appearing in many different contexts will continue to do so, as opposed to those with more situational usage.

Kneser-Ney Discounting

 $ P_{AbsoluteDiscounting}(w_i|w_{i-1}) = \frac{C(w_{i-1}w_i)-d}{\sum_vC(w_{i-1}v)}+\lambda(w_{i-1})P(w_i) $

Where first term is the discounted bigram, and the second term is the unigram with
an interpolation weight λ.

Interpolated Absolute Discounting Applied to Bigrams

Absolute discounting is a method of discounting the counts for n-grams. In absolute discounting, a fixed (absolute) discount d is subtracted from each count. The intuition is that a small discount d will not significantly affect high count estimates, which are already well-estimated, but will modify the smaller counts in a generally positive way. 

University of California, Santa Cruz

University of Michigan - Ann Arbor

Smoothing (Discounting) is the modification that keeps a language model from assigning zero probability to unseen events, by shaving off a bit of probability mass from some more frequent events and give it to the events that were never seen.

Smoothing (Discounting)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Absolute Discounting

Laplace (Add-One) Smoothing is a smoothing algorithm that adds one to all the bigram counts, before normalizing them into probabilities, hence all counts that were zero are now one, and so on. This algorithm provides a useful baseline, but does not perform well enough to be used in modern n-gram models.

Laplace (Add-One) Smoothing

Backoff is the method of estimating probabilities using to a lower-order n-gram if we have zero evidence for a higher-order n-gram; if the n-gram we need has zero counts, we approximate it by backing off to the (n-1)-gram, and continue backing off until we reach a history that has some counts.

Learn Before

Related

Learn After