Google

In the chain rule for sequence probability, the base case is the first token, $x_0$. Since there are no preceding tokens, its probability is its marginal probability, $\text{Pr}(x_0)$. In many language models, this initial token is a deterministic start-of-sequence symbol, meaning its probability is fixed at 1, i.e., $\text{Pr}(x_0) = 1$. This assumption simplifies the joint probability calculation for the rest of the sequence. Specifically, the probability of the sequence following the initial token, $\text{Pr}(x_1, ..., x_m|x_0)$, is unaffected when multiplied by $\text{Pr}(x_0)$, as shown by the equation: $$ \text{Pr}(x_0) \text{Pr}(x_1, ..., x_m|x_0) = \text{Pr}(x_1, ..., x_m|x_0) $$ This effectively means the calculation can start from the second token, conditioned on the first.

Base Case for Sequence Probability

The log-likelihood of a sequence $\mathbf{x} = (x_0, \dots, x_m)$ is computed by summing the log-probabilities of predicting each token given its predecessors. This method is derived from the chain rule of probability. The full expression is $\log \text{Pr}(\mathbf{x}) = \log \text{Pr}(x_0) + \sum_{j=1}^{m} \log \text{Pr}(x_j|\mathbf{x}_{<j})$. For practical training purposes, the probability of the initial token, $\text{Pr}(x_0)$, is often assumed to be 1 (making its log-probability 0), especially when it's a fixed start-of-sequence symbol. This simplifies the objective to summing only the conditional log-probabilities for the remaining tokens: $$ \mathcal{L}_{\theta}(\mathbf{x}) = \sum_{j=1}^{m} \log \text{Pr}_{\theta}(x_j|\mathbf{x}_{<j}) $$ In short, the process involves calculating the token prediction log-probability at each position in the sequence and then adding these values together.

Log-Likelihood Objective for Language Model Training

A language model calculates the joint probability of a sequence of tokens `(x_0, x_1, ..., x_m)`. The first token, `x_0`, is a special, deterministic start-of-sequence symbol. How does the nature of this specific first token typically affect the overall calculation of the sequence's joint probability?

A language model is calculating the probability for the three-token sequence `(x_0, x_1, x_2)`. The first token, `x_0`, is a fixed, deterministic start-of-sequence symbol. The model provides the following conditional probabilities: `Pr(x_1 | x_0) = 0.2` and `Pr(x_2 | x_0, x_1) = 0.5`. What is the joint probability of the entire sequence, `Pr(x_0, x_1, x_2)`? Explain your reasoning.

Calculating Sequence Probability with a Start Token

Based on the information provided in the case study, calculate the probability of the first token, `Pr(<s>)`. Explain what this specific probability value implies about the nature and function of the `<s>` token within this model's architecture.

Learn Before

Related