A team is optimizing a language model to handle extremely long text sequences. To reduce the computational workload, they modify the attention mechanism so that for any new token being generated, its output is calculated based on only a small, fixed subset of the preceding tokens. Which statement best evaluates why this change is unlikely to solve the primary memory consumption issue during the generation of very long sequences?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is optimizing a language model to handle extremely long text sequences. To reduce the computational workload, they modify the attention mechanism so that for any new token being generated, its output is calculated based on only a small, fixed subset of the preceding tokens. Which statement best evaluates why this change is unlikely to solve the primary memory consumption issue during the generation of very long sequences?
Sparse Attention: Computation vs. Memory
By modifying a language model's attention mechanism to only calculate scores for a small subset of previous tokens (sparse computation), the memory footprint required for storing the historical key and value vectors for all preceding tokens is also proportionally reduced.