Notation for a New Value Vector (v_i')
The notation represents a new value vector associated with the index . In the context of transformer models during inference, this vector is generated for a new token. It is then paired with its corresponding new key vector, , and this pair is added to the Key-Value cache. The bold font for signifies that it is a vector quantity.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Applying Vector Notation in Autoregressive Generation
During a single step of autoregressive generation in a transformer-based model, a new query vector (
q') and a new key vector (k') are computed for the token being generated. What is the immediate role of these two new vectors in the self-attention mechanism?During a single step of autoregressive generation for a new token, the newly computed value vector (
v') is immediately used to calculate the output for that same token.Notation for a New Value Vector (v_i')
Learn After
During the process of generating text one token at a time, a model computes a new vector denoted as Vi' for the most recent token. To efficiently generate the next token, what is the primary, immediate purpose of this Vi' vector?
Role of the New Value Vector in Inference
In a transformer model generating text token-by-token, the newly computed value vector for the current position i', denoted as Vi', is immediately weighted by attention scores to produce the output for that same position i'.