Google

In autoregressive models, the selection of the next token, $\bar{y}_i$, is formally represented as drawing a sample from the model's conditional probability distribution. This is expressed by the formula: $$\bar{y}_i \sim \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$$ This notation signifies that the token $\bar{y}_i$ is sampled from the probability distribution over all possible tokens $y_i$, conditioned on the input context $\mathbf{x}$ and the sequence of previously generated tokens $\mathbf{y}_{<i}$. The context of preceding tokens, $\mathbf{y}_{<i}$, is sometimes written more compactly as $\mathbf{y}_i$.

Formula for Token Sampling in Autoregressive Models

In autoregressive text generation, a new sequence is formed at each step by appending a newly generated token to the existing sequence. If a token $\bar{y}_i$ is sampled at step $i$, the new sequence, denoted $\bar{\mathbf{y}}$ (or sometimes `y¯`), is constructed by concatenating the preceding sequence $y_1...y_{i-1}$ with this sampled token. This process is formally represented by the equation: $$\bar{\mathbf{y}} = y_1...y_{i-1}\bar{y}_i$$

Sequence Extension with a Sampled Token

An autoregressive language model has generated the sequence of tokens: 'The quick brown fox'. It is now about to generate the next token. Which expression accurately describes how the model will select this *next* token?

An autoregressive model selects the next token by sampling from a conditional probability distribution, represented by the formula: $$\bar{y}_i \sim Pr(y_i|x, y_{<i})$$ Match each component of this formula to its correct description.

Based on the formal expression for token selection, $$\bar{y}_i \sim \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$$, explain why generating different outputs from the same initial text is the expected behavior for this type of model.

Learn Before

Related