Google

In a causal self-attention mechanism, a query at a given position `i` can only interact with keys at positions `j` where $j \le i$. For a sequence of length 7 (indexed 0 to 6), this results in the following explicit set of query-key dot products being computed:
- $q_0$: $q_0 k_0^T$
- $q_1$: $q_1 k_0^T, q_1 k_1^T$
- $q_2$: $q_2 k_0^T, q_2 k_1^T, q_2 k_2^T$
- $q_3$: $q_3 k_0^T, q_3 k_1^T, q_3 k_2^T, q_3 k_3^T$
- $q_4$: $q_4 k_0^T, q_4 k_1^T, q_4 k_2^T, q_4 k_3^T, q_4 k_4^T$
- $q_5$: $q_5 k_0^T, q_5 k_1^T, q_5 k_2^T, q_5 k_3^T, q_5 k_4^T, q_5 k_5^T$
- $q_6$: $q_6 k_0^T, q_6 k_1^T, q_6 k_2^T, q_6 k_3^T, q_6 k_4^T, q_6 k_5^T, q_6 k_6^T$

Explicit Enumeration of Causal Self-Attention Dot Products

In a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`, how many total query-key dot product computations are performed for an input sequence of length 5 (indexed 0 to 4)?

In a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`, list all the query-key dot products that are computed for the query vector at position 4 (`q_4`) in a sequence of length 8 (indexed 0 to 7).

Causal Self-Attention Dot Product Enumeration

For a sequence of 10 tokens (indexed 0 to 9), which of the following query-key dot product computations would be invalid in a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`?

Learn Before

Related