Google

The notation `Pr([x, y])` represents the probability that a language model will generate the specific token sequence formed by concatenating an input sequence `x` and an output sequence `y`. This is also known as the joint probability of the sequences `x` and `y`.

Probability of a Concatenated Token Sequence

A fundamental difference exists between standard language modeling and supervised fine-tuning objectives. Standard language modeling minimizes the loss over all tokens of a concatenated input-output sequence $$\mathrm{seq}_{\mathbf{x},\mathbf{y}} = [\mathbf{x},\mathbf{y}]$$, optimizing the joint log-probability $$\log \mathrm{Pr}_{\theta}(\mathbf{x},\mathbf{y})$$. In contrast, fine-tuning focuses on the conditional log-probability of the output. By applying the chain rule, the joint sequence probability is decomposed into the probability of the input and the conditional probability of the output: $$\log \mathrm{Pr}_{\theta}(\mathrm{seq}_{\mathbf{x},\mathbf{y}}) = \log \mathrm{Pr}_{\theta}(\mathbf{x}) + \log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$$. In the fine-tuning context, the loss computation over the input $$\mathbf{x}$$ is set to $${}0$$, meaning the loss is computed exclusively for the output tokens $$\mathbf{y}$$.

Conditional vs. Joint Probability Objectives in Language Modeling

The relationship between the joint, conditional, and marginal probabilities of token sequences is defined by the chain rule of probability. In logarithmic form, this allows the conditional log-probability of an output sequence `y` given an input `x` to be expressed in terms of the joint and marginal log-probabilities. The formula is: $$ \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) = \log \text{Pr}_{\theta}([\mathbf{x}, \mathbf{y}]) - \log \text{Pr}_{\theta}(\mathbf{x}) $$ This identity can be rearranged to show how the joint log-probability of the concatenated sequence `[x, y]` is composed of the marginal and conditional parts: $$ \log \text{Pr}_{\theta}([\mathbf{x}, \mathbf{y}]) = \log \text{Pr}_{\theta}(\mathbf{x}) + \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) $$ This relationship is fundamental for training language models, where the objective is often to maximize the conditional log-probability `log Pr_θ(y|x)`.

Relationship Between Joint, Conditional, and Marginal Log-Probabilities of Sequences

A general and foundational approach to language modeling involves treating an input `x` and an output `y` as a single, concatenated sequence. The training objective in this framework is to model the joint log-probability of the sequence, `log Prθ(x, y)`. This is accomplished by minimizing a loss function that is calculated over all tokens in the combined sequence, `[x, y]`, based on the chain rule of probability.

General Language Modeling Objective based on Joint Log-Probability

A language model is being used to determine the likelihood of a specific sentence. Let the input sequence `x` be 'The sun is' and the output sequence `y` be 'shining brightly'. The notation `Pr([x, y])` represents the probability of the model generating the full, combined sequence. Which statement best analyzes what this probability value signifies?

Consider two distinct, non-empty token sequences, `x` and `y`. A language model calculates the probability of the combined sequence `[x, y]` as `Pr([x, y])` and the probability of the reverse combined sequence `[y, x]` as `Pr([y, x])`. Would a well-trained language model typically assign an equal probability to these two combined sequences (i.e., is `Pr([x, y]) = Pr([y, x])` generally true)? Justify your answer based on how language models process sequential information.

Analysis of Sequence Order on Joint Probability

The conditional log-probability of an output sequence `y` given an input sequence `x`, denoted `log Pr(y|x)`, can be calculated using the joint log-probability of the concatenated sequence `[x, y]` and the marginal log-probability of the input sequence `x`. The relationship is defined by the formula: $$ \log \text{Pr}(\mathbf{y}|\mathbf{x}) = \log \text{Pr}([\mathbf{x}, \mathbf{y}]) - \log \text{Pr}(\mathbf{x}) $$ This equation is derived by taking the logarithm of the standard definition of conditional probability, $Pr(y|x) = Pr([x,y]) / Pr(x)$, and is crucial for connecting different probabilistic objectives in language modeling.

Conditional Log-Probability via Joint and Marginal Log-Probabilities

A researcher is comparing two language models, Model A and Model B, by evaluating the joint probability they assign to complete sentences. The input sequence `x` is 'The old library held...'. The researcher considers two possible output sequences: `y1` = '...a secret.' and `y2` = '...many books.'. The models produce the following joint probabilities for the concatenated sequences `[x, y]`:

Learn Before

Related