Case Study

Evaluating Model Prediction Quality

A language model is being trained to complete the sentence 'The sky is...'. The training data indicates the correct next word is 'blue'. The model's performance is measured by a value that quantifies the difference between its predicted probabilities and the correct outcome. A lower value indicates a better prediction.

Two different versions of the model produce the following probability distributions for the next word:

  • Model Alpha: Assigns a probability of 0.8 to 'blue', 0.1 to 'green', and 0.1 to 'cloudy'.
  • Model Beta: Assigns a probability of 0.3 to 'blue', 0.5 to 'cloudy', and 0.2 to 'green'.

Which model, Alpha or Beta, would be assigned a lower performance penalty (i.e., a lower loss value) for this specific prediction? Justify your reasoning.

0

1

Updated 2025-10-03

Contributors are:

Who are from:

Tags

Ch.1 Pre-training - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Application in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science