Attention Weight Matrix (α)

Sparse Attention

Self-attention layers' first approach

In a general attention mechanism, the output is calculated as a weighted sum of the Value vectors, where the weights are determined by the interaction between Query and Key vectors. The standard formula is: $Att(\textbf{Q}, \textbf{K}, \textbf{V}) = \alpha(\textbf{Q}, \textbf{K})\textbf{V}$. Consider a scenario where this formula is mistakenly altered to be: $Att_{modified} = \alpha(\textbf{Q}, \textbf{K})\textbf{K}$. What is the most significant consequence of this modification?

Dimensional Analysis of the Attention Formula

Applying the Attention Mechanism Roles

Self-Attention Output Formula for a Single Query

In a self-attention mechanism, the raw attention scores (β) for a single query vector with respect to three key vectors are calculated as [2.0, 1.0, 0.5]. To convert these scores into a probability distribution, a normalization function is applied. What is the resulting normalized attention weight (α) corresponding to the first key vector (score of 2.0)?

In a self-attention mechanism, a set of raw, unnormalized attention scores for a specific query are [1.5, 0.5, -1.0]. If a constant value of 10 is added to each of these scores, resulting in a new set of scores [11.5, 10.5, 9.0], how will the final normalized attention weights (the probability distribution) calculated from the new scores compare to the weights calculated from the original scores?

Calculating and Interpreting Attention Weights

Self-Attention Output Formula for a Single Query

Computing Attention Weights in Sequence Parallelism

Distributed Attention Weight Formula