Learn Before
Auto-Regressive Generation Process
When applying a language model, the generation of text typically follows an autoregressive process. At each individual step, the model takes a token, denoted as , as its current input. It then predicts the subsequent token, , by selecting the one that maximizes the conditional probability . This ensures each new prediction is conditioned on the sequence of all previously generated tokens.
0
1
References
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.5 Inference - Foundations of Large Language Models
Related
Probability Distribution Formula for an Encoder-Softmax Language Model
Auto-Regressive Generation Process
Formal Definition of LLM Inference
Model Parameterization by θ
A language model built with a deep neural network is given the input sequence 'The cat sat on the'. The model's vocabulary consists of the following tokens: {a, cat, hat, mat, on, sat, the}. What does the model produce as its immediate, direct output to predict the very next token?
Analyzing Language Model Outputs
Explaining Language Model Output Behavior
Learn After
Token Selection from Probability Distribution
Step-by-Step Example of Auto-Regressive Sequence Generation
Mathematical Formulation of Draft Model Prediction in Speculative Decoding
Iterative Context Update in Autoregressive Generation
Key-Value (KV) Cache in Transformer Inference
Sequential Generation of Output Tokens
Context Shifting in Auto-Regressive Generation
A language model is generating a sentence and has so far produced the sequence:
['The', 'cat', 'sat']. Based on the principles of sequential, one-at-a-time token generation where each new token depends on the ones before it, what is the direct input the model will use to determine the next token in the sequence?A language model generates text by producing a single token at each step, using the entire sequence generated so far as the context for the next token. Arrange the following events in the correct chronological order to illustrate the generation of two new tokens following the initial input 'The ocean is'.
A researcher develops a novel text generation model. Given an input like 'The movie was', instead of generating one token at a time, this model predicts the entire completion (e.g., 'incredibly boring and predictable') in a single, parallel step. Which core principle of the standard auto-regressive process is fundamentally violated by this new model's design?