Learn Before
Span Prediction Loss Formula
The loss for a span prediction task is calculated as the average negative log-likelihood of the predicted probabilities for the start and end positions of the answer span. The formula is: Where: - is the number of tokens in the context text. - is the model's predicted probability that token is the start of the answer span. - is the model's predicted probability that token is the end of the answer span.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.1 Pre-training - Foundations of Large Language Models
Related
Span Prediction Loss Formula
A question-answering model is being trained to identify a specific answer span within a passage. The model's training objective is to minimize a loss calculated from two separate predictions for each token: the probability of it being the start of the answer and the probability of it being the end. The total loss is calculated by summing the negative log-likelihoods from both prediction networks. In which of the following scenarios would the model incur the highest training loss for a single training example?
Analyzing Span Prediction Model Loss
Rationale for Combined Span Prediction Loss
Learn After
A language model is processing a single training example for a question-answering task. The correct answer span begins at token 25 and ends at token 28. The model predicts the probability of token 25 being the start as 0.6, and the probability of token 28 being the end as 0.7. Using the standard loss calculation for this task, which sums the negative log-likelihoods of the correct start and end positions (
Loss = - (log p_start + log p_end)), what is the loss value for this example? (Use the natural logarithm, ln, and round to three decimal places).Comparing Model Performance via Loss Calculation
Analysis of Span Prediction Loss