Google

In autoregressive models, a new set of vectors is generated for the current token at position `i'`: a query vector (`q'` or `q_{i'}`), a key vector (`k'` or `k_{i'}`), and a value vector (`v'` or `v_{i'}`). The new query `q'` interacts with all previous key vectors to compute attention scores. The new key `k'` and value `v'` are then appended to the Key-Value cache, making them available for subsequent tokens.

Notation for Current Query, Key, and Value Vectors (`q'`, `k'`, `v'`)

The notation $\mathbf{V}_{i'}$ represents a new value vector associated with the index $i'$. In the context of transformer models during inference, this vector is generated for a new token. It is then paired with its corresponding new key vector, $\mathbf{k}_{i'}$, and this pair is added to the Key-Value cache. The bold font for $\mathbf{V}$ signifies that it is a vector quantity.

Notation for a New Value Vector (V_i')

Based on the scenario below, explain the specific roles of the newly created `q'` and `k'` vectors in the model's process of determining the *next* token.

Applying Vector Notation in Autoregressive Generation

During a single step of autoregressive generation in a transformer-based model, a new query vector (`q'`) and a new key vector (`k'`) are computed for the token being generated. What is the immediate role of these two new vectors in the self-attention mechanism?

During a single step of autoregressive generation for a new token, the newly computed value vector (`v'`) is immediately used to calculate the output for that same token.

Learn Before

Related