Learn Before
Per-Token Time Complexity Across Layers in Self-Attention
During autoregressive generation, calculating self-attention for a single new token over a context sequence of length len has a linear time complexity of across transformer layers. This cost arises because at each layer, the two primary matrix-vector operations—the dot products between the query and previous key vectors, and the weighted summation of previous value vectors—both scale linearly with len.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Time Complexity of Self-Attention in Autoregressive Generation
In a model that generates text one token at a time, suppose it has already produced a sequence of length
Nand is now calculating the next token (at positionN+1). Which of the following best identifies the two primary computational operations within the attention mechanism that cause the cost of this single step to scale linearly with the current sequence lengthN?Analyzing Generation Latency
Predicting Attention Computation Time
Per-Token Time Complexity Across Layers in Self-Attention