Log-Likelihood of a Sequence
The log-likelihood of a sequence of tokens, denoted as , is a fundamental metric used in training and evaluating language models. It is calculated by applying the logarithm to the chain rule of probability, which transforms the product of conditional probabilities into a more numerically stable sum. For a sequence , the log-likelihood is computed as: This calculation is performed for each sequence within a training dataset to determine the overall likelihood of the data given the model.
0
1
References
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.5 Inference - Foundations of Large Language Models
Related
Log-Likelihood of a Sequence
When calculating the probability of a long sequence of words, the standard approach involves multiplying many conditional probabilities, each of which is a value between 0 and 1. This product is often converted into a sum by applying the logarithm to each term. What is the primary computational reason for this transformation?
A language model calculates the probability of a sequence of tokens, , using the product of conditional probabilities: . To improve numerical stability and simplify calculations, this product is converted into a sum by taking the logarithm. Which of the following expressions correctly represents the log-probability, ?
Calculating Sequence Log-Probability
Learn After
Log-Probability of a Ranked Sequence
Log-Likelihood Objective for Language Model Training
A language model is generating a sequence of tokens. It has computed the following conditional log-probabilities for a three-token sequence, where each token's probability is dependent on the ones that came before it:
- Log-probability of the first token: -1.8
- Log-probability of the second token, given the first: -2.5
- Log-probability of the third token, given the first two: -1.2
Based on these values, what is the total log-likelihood of this entire three-token sequence?
Evaluating Sentence Plausibility
A language model has calculated the total log-likelihood for the sequence of tokens: ["The", "quick", "brown", "fox"]. The calculation involves summing the conditional log-probabilities of each token given the preceding ones. If the third token is changed from "brown" to "lazy", creating the new sequence ["The", "quick", "lazy", "fox"], which set of conditional log-probabilities must be re-calculated to find the new total log-likelihood?
Applying Log-Likelihood Calculation to a Training Dataset