A team is optimizing a language model to handle extremely long text sequences. To reduce the computational workload, they modify the attention mechanism so that for any new token being generated, its output is calculated based on only a small, fixed subset of the preceding tokens. Which statement best evaluates why this change is unlikely to solve the primary memory consumption issue during the generation of very long sequences?

Sparse Attention: Computation vs. Memory

By modifying a language model's attention mechanism to only calculate scores for a small subset of previous tokens (sparse computation), the memory footprint required for storing the historical key and value vectors for all preceding tokens is also proportionally reduced.