An Intrinsic Evaluation metric is one that measures the quality of a model independent of any application, with the needing of a test corpus. Given a corpus of text, we divide the data into training and test sets, and train the parameters of both models on the training set. Whichever model assigns a high probability to the test set, meaning it more accurately predicts the test set, is a better model.

University of Michigan - Ann Arbor

Models are examined by how well they predict unseen text. Good models assign higher probabilities to unseen data.

Evaluating language models

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

An Extrinsic Evaluation is an end-to-end evaluation of the performance of a language model that embeds it in an application and measure how much the application improves. This is the only way to know if a particular improvement in a component is really going to help the task at hand.


Extrinsic Evaluation

Intrinsic Evaluation

Using perplexity is a method of measuring quality of a language model. Perplexity of a model θ on an unseen test set is the inverse probability assigned by θ to the test set, which is normalized by the test set length.

Minimizing perplexity is equivalent to maximizing the test set probability in the language model.

Learn Before

Related