Learn Before
Conditional Log-Probability via Joint and Marginal Log-Probabilities
The conditional log-probability of an output sequence y given an input sequence x, denoted log Pr(y|x), can be calculated using the joint log-probability of the concatenated sequence [x, y] and the marginal log-probability of the input sequence x. The relationship is defined by the formula: This equation is derived by taking the logarithm of the standard definition of conditional probability, , and is crucial for connecting different probabilistic objectives in language modeling.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Conditional vs. Joint Probability Objectives in Language Modeling
Relationship Between Joint, Conditional, and Marginal Log-Probabilities of Sequences
General Language Modeling Objective based on Joint Log-Probability
A language model is being used to determine the likelihood of a specific sentence. Let the input sequence
xbe 'The sun is' and the output sequenceybe 'shining brightly'. The notationPr([x, y])represents the probability of the model generating the full, combined sequence. Which statement best analyzes what this probability value signifies?Analysis of Sequence Order on Joint Probability
Conditional Log-Probability via Joint and Marginal Log-Probabilities
Model Comparison Using Joint Sequence Probability
Learn After
A language model is given an input sequence, x. The model calculates the log-probability of this input sequence as -12.5. The model then generates a completion, y, and calculates the log-probability of the full, combined sequence [x, y] as -15.0. Based on these values, what is the conditional log-probability of the model generating the completion y given the input x?
Comparing Language Model Outputs
Calculating Joint Log-Probability