Learn Before
Comparing Model Performance via Loss Calculation
You are evaluating two different language models, Model A and Model B, on a question-answering task. For a single training example, the correct answer span begins at token 15 and ends at token 18. The models produce the following probabilities for the correct start and end tokens:
- Model A: Probability of token 15 being the start = 0.8; Probability of token 18 being the end = 0.7.
- Model B: Probability of token 15 being the start = 0.5; Probability of token 18 being the end = 0.4.
Using the loss calculation for a single example, Loss = - (log p_start + log p_end), determine which model is performing better on this specific example. Justify your answer by calculating and comparing the loss for each model. (Use the natural logarithm, ln, for your calculations).
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is processing a single training example for a question-answering task. The correct answer span begins at token 25 and ends at token 28. The model predicts the probability of token 25 being the start as 0.6, and the probability of token 28 being the end as 0.7. Using the standard loss calculation for this task, which sums the negative log-likelihoods of the correct start and end positions (
Loss = - (log p_start + log p_end)), what is the loss value for this example? (Use the natural logarithm, ln, and round to three decimal places).Comparing Model Performance via Loss Calculation
Analysis of Span Prediction Loss