Google

The fundamental computational task of an autoregressive language model is to model the conditional probability of the next token given the preceding context, `Pr(y_i|x, y_{<i})`. A key requirement for practical implementation is that this computation must be performed in an efficient manner.

Core Computational Task in Autoregressive Generation

An autoregressive language model is in the process of generating a response. It has so far produced the token sequence: `['The', 'quick', 'brown']`. To determine the *very next* token, what is the primary probability distribution the model must compute?

A development team is designing a new text-generation model and is considering two different computational strategies. Analyze these strategies and determine which one correctly implements the core task of an autoregressive model. Justify your choice by explaining the fundamental principle that the correct strategy upholds.

Evaluating Language Model Generation Strategies

An autoregressive language model generates text one token at a time, where each new token's probability depends on all the tokens that came before it. Based on this sequential dependency, explain the primary challenge this process poses for achieving high computational parallelism during the text generation (or 'inference') phase.

Computational Constraints in Autoregressive Generation

To generate a sequence of text, the fundamental computational step for an autoregressive model is to calculate the joint probability of all potential future tokens at once, conditioned only on the initial prompt.

Learn Before

Related