Based on the provided log-probabilities, which model's completion (`y_A` or `y_B`) is more probable given the input prompt `x`? Justify your answer by calculating the conditional log-probability for each model's output.

Google

The conditional log-probability of an output sequence `y` given an input sequence `x`, denoted `log Pr(y|x)`, can be calculated using the joint log-probability of the concatenated sequence `[x, y]` and the marginal log-probability of the input sequence `x`. The relationship is defined by the formula: $$ \log \text{Pr}(\mathbf{y}|\mathbf{x}) = \log \text{Pr}([\mathbf{x}, \mathbf{y}]) - \log \text{Pr}(\mathbf{x}) $$ This equation is derived by taking the logarithm of the standard definition of conditional probability, $Pr(y|x) = Pr([x,y]) / Pr(x)$, and is crucial for connecting different probabilistic objectives in language modeling.

Conditional Log-Probability via Joint and Marginal Log-Probabilities

A language model is given an input sequence, **x**. The model calculates the log-probability of this input sequence as -12.5. The model then generates a completion, **y**, and calculates the log-probability of the full, combined sequence **[x, y]** as -15.0. Based on these values, what is the conditional log-probability of the model generating the completion **y** given the input **x**?

Comparing Language Model Outputs

A language model is evaluating a potential completion for a given prompt. The model determines that the conditional log-probability of the completion (`y`) given the prompt (`x`) is -4.5. It also calculates the marginal log-probability of the prompt (`x`) itself to be -9.2. Based on this information, what is the joint log-probability of the combined sequence `[x, y]`?

Learn Before

Related