Updating the KV Cache

In a self-attention layer processing an input sequence of two tokens, let the input vector for the first token be x_1 and for the second token be x_2. The layer generates a query vector q_1 (for the first token) and a key vector k_2 (for the second token). Which statement accurately describes the relationship between these inputs and generated vectors?

Correcting a Misconception in Vector Generation

Calculating a Query Vector in Self-Attention

In a standard self-attention mechanism, an input vector is transformed into three separate vectors (Query, Key, and Value) using three distinct, learned weight matrices. Imagine a modified self-attention layer where these three weight matrices are constrained to be identical. What would be the most direct consequence of this change?

Single-Step Autoregressive Generation with a Key-Value (KV) Cache

Formula for Updating the Key Matrix in the KV Cache

Formula for Updating the Value Matrix in the KV Cache

Example of a Single-Step KV Cache Update

During autoregressive text generation, a model has already processed N tokens and stored their corresponding key and value vectors in a cache. When the model processes the (N+1)-th token, how is this cache utilized and modified to compute the output for this new step?

An autoregressive model is generating a sequence and has just processed the token at position t. The Key-Value cache currently stores the key and value vectors for all tokens from position 1 to t. As the model processes the next token at position t+1, which statement correctly describes how the cache is updated and used for the attention calculation at this new step?

Notation for Current Query, Key, and Value Vectors (q', k', v')

Diagram of a Single-Step KV Cache Update and Attention

Debugging a Flawed KV Cache Implementation

Single-Step Autoregressive Generation with a Key-Value (KV) Cache

Diagram of the Decoding Phase

Comparison of Prefilling and Decoding Phases

Disaggregation of Prefilling and Decoding using Pipelined Engines

After a large language model processes an initial prompt, it enters a generation stage where it produces the output sequence one token at a time. In each step of this stage, a new query vector is generated for the current position, and it must perform an attention operation over the key-value pairs of the initial prompt plus all the key-value pairs of the tokens generated in previous steps. As the output sequence gets longer, what becomes the most significant performance bottleneck for generating each new token?

A large language model has finished processing an initial prompt and is about to generate the first token of its response. Arrange the following events in the correct chronological order for this single generation step.

Evaluating an Inference Optimization Proposal

You run an internal LLM inference service for empl...

You’re on-call for an internal LLM chat service. M...

You operate a GPU-backed LLM service that uses con...

Your company’s internal LLM service handles many c...

Evaluating a serving design that combines prefix caching with paged KV memory under mixed prompt lengths

Choosing a KV-cache strategy for shared-prefix traffic under GPU memory pressure

Diagnosing and Redesigning KV-Cache Memory Behavior in a Multi-Tenant LLM Serving Stack

Stabilizing latency and GPU memory in a chat-completions service with shared system prompts

Root-cause and mitigation plan for OOMs and latency spikes during shared-prefix, long-generation traffic

Post-incident analysis: KV-cache growth, fragmentation, and shared-prefix reuse in a streaming LLM service

Decoding Phase Goal Formula

Single-Step Autoregressive Generation with a Key-Value (KV) Cache