The best performing version of Kneser-Ney smoothing is called Modified Kneser-Ney Smoothing. Rather than use a single fixed discount d, modified Kneser-Ney uses three different discounts $d_1, d_2, and d_3+$ for n-grams with counts of 1, 2 and three or more, respectively.

University of Michigan - Ann Arbor

Kneser-Ney discounting augments absolute discounting to better handle low-order unigram distributions. Rather than estimating the probability of a word w simply appearing, we create a unigram model that estimates the probability of w appearing as a novel continuation, completing some previously unseen bigram. This estimate is based on the number of bigrams w has previously completed, with the intuition that words appearing in many different contexts will continue to do so, as opposed to those with more situational usage.

Kneser-Ney Discounting

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

 $P_{KN}(w_i|w_{i-1}) = \frac{max(C(w_{i-1}w_i) - d, 0)}{C(w_{i-1})}+\lambda(w_{i-1})P_{CONTINUATION}(w_i)$

Where λ is a normalizing constant to distribute probability mass:

$\lambda(w_{i-1}) = \frac{d}{\sum_vC(w_{i-1}v)}|\{w:C(w_{i-1}w)>0\}|$

Learn Before

Related