conditional probability for the next word: $P(w_n|w_{1:n-1}) \approx P(w_n|w_{n-N+1:n-1})$
probability of a complete word sequence: $P(w_{1:n}) \approx \Pi^n_{k=1}P(w_k|w_{k-N+1:k-1})$


University of Illinois at Urbana-Champaign

To estimate the n-gram probablities, we get estimation normalize the MLE estimate for the parameters of an n-gram model by getting counts from a corpus, and normalizing the counts so that they lie between 0 and 1：
$P(w_n|w_{n-N+1:n-1}) = \frac{C(w_{n-N+1:n-1} wn)}{C(w_{n-N+1:n-1}}$

MLE & Normalizing

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Calculated by dividing the observed frequency of a particular sequence by the observed frequency of a prefix. It is used as a way to estimate probabilities is an example of maximum likelihood estimation or MLE.

Relative frequency

By using log probabilities instead of raw probabilities, we get numbers that are not as small. The result of doing all computation and storage in log space is that we only need to convert back into probabilities if we need to report them at the end; then we can just take the exp of the logprob:
$p_1 \times p_2 \times p_3 \times p_4 = exp( log{p_1} + log{p_2} + log{p_3}+ log{p_4})$

Learn Before

Related