Log-Probability of a Ranked Sequence

Log-Likelihood Objective for Language Model Training

A language model is generating a sequence of tokens. It has computed the following conditional log-probabilities for a three-token sequence, where each token's probability is dependent on the ones that came before it:

• Log-probability of the first token: -1.8
• Log-probability of the second token, given the first: -2.5
• Log-probability of the third token, given the first two: -1.2

Based on these values, what is the total log-likelihood of this entire three-token sequence?

Evaluating Sentence Plausibility

A language model has calculated the total log-likelihood for the sequence of tokens: ["The", "quick", "brown", "fox"]. The calculation involves summing the conditional log-probabilities of each token given the preceding ones. If the third token is changed from "brown" to "lazy", creating the new sequence ["The", "quick", "lazy", "fox"], which set of conditional log-probabilities must be re-calculated to find the new total log-likelihood?

Applying Log-Likelihood Calculation to a Training Dataset