Google

This formula calculates the output for a single attention head, `head_j`, in a transformer model that implements Grouped-Query Attention (GQA) with causal masking. The formula is:

$$\text{head}_j = \text{Att}_{\text{qkv}}(\mathbf{q}_i^{[j]}, \mathbf{K}_{\le i}^{[g(j)]}, \mathbf{V}_{\le i}^{[g(j)]})$$

In this equation, $$\text{Att}_{\text{qkv}}$$ represents the attention function, and $$\mathbf{q}_i^{[j]}$$ is the query vector for the current token `i` and head `j`. The keys $$\mathbf{K}$$ and values $$\mathbf{V}$$ are shared among a group of query heads, as determined by the function $$g(j)$$. The subscript $${\le i}$$ signifies that the attention is causal, meaning it only considers tokens up to the current position `i`.

Attention Head Output with Grouped Queries and Causal Masking

Using the provided case study and the attention formula, identify the specific set of Key and Value vectors that the query vector `$\mathbf{q}_5^{[8]}$` will interact with. Justify your answer by explaining how the indices `i`, `j`, and the function `g(j)` from the formula determine this outcome.

Applying Grouped-Query Attention with Causal Masking

The formula `$\text{head}_j = \text{Att}_{\text{qkv}}(\mathbf{q}_i^{[j]}, \mathbf{K}_{\le i}^{[g(j)]}, \mathbf{V}_{\le i}^{[g(j)]})$` describes the output of an attention head. Analyze the distinct roles of the `${\le i}$` subscript and the `$g(j)$` mapping function. How do these two components respectively influence the model's capabilities and computational performance?

Deconstructing the GQA Formula with Causal Masking

A transformer model calculates the output for a single attention head `j` at token position `i` using the formula: `$\text{head}_j = \text{Att}_{\text{qkv}}(\mathbf{q}_i^{[j]}, \mathbf{K}_{\le i}^{[g(j)]}, \mathbf{V}_{\le i}^{[g(j)]})$`, where `g(j)` maps the query head `j` to a specific group. What is the primary consequence of using the group-indexed key `$\mathbf{K}_{\le i}^{[g(j)]}$` instead of a head-specific key `$\mathbf{K}_{\le i}^{[j]}$`?

Learn Before

Related