Absolute discounting is a method of discounting the counts for n-grams. In absolute discounting, a fixed (absolute) discount d is subtracted from each count. The intuition is that a small discount d will not significantly affect high count estimates, which are already well-estimated, but will modify the smaller counts in a generally positive way. 

Absolute Discounting

Laplace (Add-One) Smoothing is a smoothing algorithm that adds one to all the bigram counts, before normalizing them into probabilities, hence all counts that were zero are now one, and so on. This algorithm provides a useful baseline, but does not perform well enough to be used in modern n-gram models.

Laplace (Add-One) Smoothing

Backoff is the method of estimating probabilities using to a lower-order n-gram if we have zero evidence for a higher-order n-gram; if the n-gram we need has zero counts, we approximate it by backing off to the (n-1)-gram, and continue backing off until we reach a history that has some counts.

Backoff

Smoothing (Discounting) is the modification that keeps a language model from assigning zero probability to unseen events, by shaving off a bit of probability mass from some more frequent events and give it to the events that were never seen.

University of Michigan - Ann Arbor

For any n-gram that occurred a sufficient number of times, we might have a good estimate of its probability. But because any corpus is limited, some perfectly acceptable English word sequences are bound to be missing from it. These zero-probability n-grams raise problems if they occur in the test set:
- The model is underestimating the probability of all sorts of words that might occur.
- If the probability of any word in the test set is 0, the entire probability of the test set is 0. As by definition, perplexity is based on the inverse probability of the test set.

Unknown Words and Problem of Sparsity

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

A Closed Vocabulary system is one in which the test set can only contain words from this lexicon, and there will be no unknown words.

Closed Vocabulary

An Open Vocabulary system is one in which we model these potential unknown words in the test set by adding a pseudo-word called /<UNK>/.


Open Vocabulary

Out of Vocabulary (OOV) words are unknown words that do not occur in the training set. The percentage of OOV words that appear in the test set is called the OOV Rate.

Out of Vocabulary (OOV)

Ways to train the probabilities of the unknown word model:
- Choose a vocabulary (word list) that is fixed in advance; convert in the training set any OOV word to the unknown word token /<UNK>/ in a text normalization step; and estimate the probabilities for /<UNK>/ from its counts just like any other regular word in the training set.
- Create a vocabulary implicitly by replacing words in the training data by /<UNK>/ based on their frequency; and estimate the probabilities for /<UNK>/ like before.

Learn Before

Related

Learn After