Memory Bottleneck from KV Cache in LLMs

An auto-regressive language model is generating text and has already produced a sequence of 100 tokens. To generate the 101st token, it must calculate self-attention. If the model stores the 'key' and 'value' vectors for the first 100 tokens, which of the following best describes the computational steps required for the self-attention mechanism at this specific step?

Optimizing Chatbot Inference Speed

Computational Cost of Autoregressive Generation