Google

In autoregressive models, the selection of the next token, $$\bar{y}_i$$, is formally represented as drawing a sample from the model's conditional probability distribution. This is expressed by the formula: $$\bar{y}_i \sim \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$$ This notation signifies that the token $$\bar{y}_i$$ is sampled from the probability distribution over all possible tokens $$y_i$$, conditioned on the input context $$\mathbf{x}$$ and the sequence of previously generated tokens $$\mathbf{y}_{<i}$$.

Formula for Token Sampling in Autoregressive Models

An autoregressive model selects the next token by sampling from a conditional probability distribution, represented by the formula: $$\bar{y}_i \sim Pr(y_i|x, y_{<i})$$ Match each component of this formula to its correct description.

Based on the formal expression for token selection, $$\bar{y}_i \sim \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$$, explain why generating different outputs from the same initial text is the expected behavior for this type of model.

Explaining Model Output Variability

In autoregressive text generation, a new sequence is formed at each step by appending a newly generated token to the existing sequence. If a token $$\bar{y}_i$$ is sampled at step $$i$$, the new sequence, denoted $$\bar{\mathbf{y}}$$, is constructed by concatenating the preceding sequence $$y_1...y_{i-1}$$ with this sampled token. This process is formally represented by the equation:

$$\bar{\mathbf{y}} = y_1...y_{i-1}\bar{y}_i$$

Formula for Sequence Extension with a Sampled Token

An autoregressive language model has generated the sequence of tokens: 'The quick brown fox'. It is now about to generate the next token. Which expression accurately describes how the model will select this *next* token?

Learn Before

Related