For a test set $W = w_1…w_n$, its perplexity is
$$PP(W) = \left(\prod_{i=1}^{N}\frac{1}{P(w_i|w_1...w_{i-1})}\right)^{\frac{1}{N}}$$

Computing Perplexity

The Branching Factor of a language is the number of possible next words that can follow any word. Perplexity can be thought of as the Weighted Average Branching Factor of a language.

Branching Factor and Perplexity

Using perplexity is a method of measuring quality of a language model. Perplexity of a model θ on an unseen test set is the inverse probability assigned by θ to the test set, which is normalized by the test set length.

Minimizing perplexity is equivalent to maximizing the test set probability in the language model.

University of Michigan - Ann Arbor

University of Texas at San Antonio

Models are examined by how well they predict unseen text. Good models assign higher probabilities to unseen data.

Evaluating language models

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

An Extrinsic Evaluation is an end-to-end evaluation of the performance of a language model that embeds it in an application and measure how much the application improves. This is the only way to know if a particular improvement in a component is really going to help the task at hand.


Extrinsic Evaluation

An Intrinsic Evaluation metric is one that measures the quality of a model independent of any application, with the needing of a test corpus. Given a corpus of text, we divide the data into training and test sets, and train the parameters of both models on the training set. Whichever model assigns a high probability to the test set, meaning it more accurately predicts the test set, is a better model.

Learn Before

Related

Learn After