Case Study

Comparing Model Performance via Loss Calculation

You are evaluating two different language models, Model A and Model B, on a question-answering task. For a single training example, the correct answer span begins at token 15 and ends at token 18. The models produce the following probabilities for the correct start and end tokens:

  • Model A: Probability of token 15 being the start = 0.8; Probability of token 18 being the end = 0.7.
  • Model B: Probability of token 15 being the start = 0.5; Probability of token 18 being the end = 0.4.

Using the loss calculation for a single example, Loss = - (log p_start + log p_end), determine which model is performing better on this specific example. Justify your answer by calculating and comparing the loss for each model. (Use the natural logarithm, ln, for your calculations).

0

1

Updated 2025-10-03

Contributors are:

Who are from:

Tags

Ch.2 Generative Models - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Analysis in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science