An autoregressive language model is in the process of generating a response. It has so far produced the token sequence: ['The', 'quick', 'brown']. To determine the very next token, what is the primary probability distribution the model must compute?

Evaluating Language Model Generation Strategies

Computational Constraints in Autoregressive Generation

To generate a sequence of text, the fundamental computational step for an autoregressive model is to calculate the joint probability of all potential future tokens at once, conditioned only on the initial prompt.