A language model generates text token by token. At each step 'i', an attention operation computes an output using a query vector and a memory component. In a standard causal implementation, this memory component is defined as the complete set of key and value vectors from all previous steps (1 to i). Based on this definition, what is the direct relationship between the size of this memory component and the length of the generated sequence 'i'?

Sparse Attention with a Fixed Key-Value Subset

Evaluating Memory Models in Attention Mechanisms

Evaluating an Attention Mechanism for a Real-Time Application