Learn Before
Notation for Current Query, Key, and Value Vectors (q', k', v')
In autoregressive models, a new set of vectors is generated for the current token at position i': a query vector (q' or q_{i'}), a key vector (k' or k_{i'}), and a value vector (v' or v_{i'}). The new query q' interacts with all previous key vectors to compute attention scores. The new key k' and value v' are then appended to the Key-Value cache, making them available for subsequent tokens.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Formula for Updating the Key Matrix in the KV Cache
Formula for Updating the Value Matrix in the KV Cache
Example of a Single-Step KV Cache Update
During autoregressive text generation, a model has already processed
Ntokens and stored their corresponding key and value vectors in a cache. When the model processes the(N+1)-th token, how is this cache utilized and modified to compute the output for this new step?An autoregressive model is generating a sequence and has just processed the token at position
t. The Key-Value cache currently stores the key and value vectors for all tokens from position 1 tot. As the model processes the next token at positiont+1, which statement correctly describes how the cache is updated and used for the attention calculation at this new step?Notation for Current Query, Key, and Value Vectors (
q',k',v')Diagram of a Single-Step KV Cache Update and Attention
Debugging a Flawed KV Cache Implementation
Single-Step Autoregressive Generation with a Key-Value (KV) Cache
Learn After
Applying Vector Notation in Autoregressive Generation
During a single step of autoregressive generation in a transformer-based model, a new query vector (
q') and a new key vector (k') are computed for the token being generated. What is the immediate role of these two new vectors in the self-attention mechanism?During a single step of autoregressive generation for a new token, the newly computed value vector (
v') is immediately used to calculate the output for that same token.Notation for a New Value Vector (v_i')