The Gaussian kernel for attention pooling can be adapted by incorporating a width parameter, denoted by $$\sigma^2$$. The modified formula is $$\alpha(\mathbf{q}, \mathbf{k}) = \exp\left(-\frac{1}{2 \sigma^2} \|\mathbf{q} - \mathbf{k}\|^2 ight)$$. Adjusting this width parameter allows control over how smoothly the attention weights decay with respect to the distance between the query $$\mathbf{q}$$ and the key $$\mathbf{k}$$.

Gaussian Attention Kernel with Variable Width

The attention scoring function (without exponentiation) derived from the Gaussian kernel is mathematically expressed as $$a(\mathbf{q}, \mathbf{k}_i) = -\frac{1}{2} \|\mathbf{q} - \mathbf{k}_i\|^2 = \mathbf{q}^	op \mathbf{k}_i -\frac{1}{2} \|\mathbf{k}_i\|^2 -\frac{1}{2} \|\mathbf{q}\|^2$$. Because the final term ($$-\frac{1}{2} \|\mathbf{q}\|^2$$) depends exclusively on the query and remains identical for all query-key pairs, it disappears completely when attention weights are normalized to $$1$$ (e.g., via the softmax operation). Furthermore, when key vectors are generated using techniques like batch or layer normalization, their norms ($$\|\mathbf{k}_i\|$$) become well-bounded and essentially constant, allowing the key-dependent term to be safely dropped without a major change in the outcome. By eliminating both of these norm terms, the Gaussian kernel conceptually simplifies into the standard dot product attention scoring function: $$a(\mathbf{q}, \mathbf{k}_i) = \mathbf{q}^	op \mathbf{k}_i$$.

Derivation of Dot Product Attention from Gaussian Kernel

The Gaussian kernel for attention pooling is defined by the formula $$\alpha(x) = \exp(-x^2 / 2)$$. It is a translation and rotation invariant kernel that assigns smoothly decaying weights to observations based on their distance from the origin.

Claude

Attention kernels $$\alpha(\mathbf{k}, \mathbf{q})$$ that are translation and rotation invariant remain unchanged in value if the key $$\mathbf{k}$$ and query $$\mathbf{q}$$ are shifted and rotated in the same manner. For simplicity, when scalar arguments $$k, q \in \mathbb{R}$$ are chosen and the key $$k = 0$$ is picked as the origin, the kernel can be expressed as a function of the query, yielding various scalar kernel functions that correspond to different notions of range and smoothness.

Translation and Rotation Invariant Attention Kernels

Dive into Deep Learning

Gaussian Attention Kernel

The boxcar kernel is an attention pooling kernel defined by the formula $$\alpha(x) = |x| < 1.0$$. It acts as an indicator function, indiscriminately attending only to observations within a distance of $$1$$ (or another defined hyperparameter) and assigning a weight of zero to all other observations.

Boxcar Attention Kernel

The constant kernel for attention pooling is defined by the formula $$\alpha(x) = 1.0 + 0 	imes x$$. It is a translation and rotation invariant kernel that assigns a uniform, constant weight of $$1.0$$ to all observations, regardless of their distance.

Constant Attention Kernel

The Epanechikov kernel for attention pooling is defined by the formula $$\alpha(x) = \max(1 - |x|, 0)$$. It is a translation and rotation invariant kernel that linearly decays to zero for observations beyond a distance of $$1$$, providing a bounded notion of range and smoothness.

Epanechikov Attention Kernel

To implement translation and rotation invariant attention kernels in practice, programmatic functions are defined for the Gaussian, Boxcar, Constant, and Epanechikov kernels. These functions take scalar distance inputs and return the corresponding attention weights ($$\alpha$$), providing distinct computational notions of range and smoothness for attention pooling operations.

Learn Before

Related

Learn After