Google

In a multi-head attention mechanism, the output of each individual attention head, denoted as $\text{head}_j$, is a vector. This vector belongs to a real-valued vector space of dimension $d_h$, which is represented by the notation: $$\text{head}_j \in \mathbb{R}^{d_h}$$

Dimensionality of an Attention Head Output

Given a representation matrix $$\mathbf{H} \in \mathbb{R}^{m \times d}$$, the multi-head self-attention function computes its output by concatenating the results from multiple individual attention heads. This relationship is formalized as:

$$F(\mathbf{H}) = \mathrm{Merge}(\mathrm{head}_1, \dots, \mathrm{head}_\tau) \mathbf{W}^{\mathrm{head}}$$

In this equation, $$\mathrm{Merge}(\cdot)$$ signifies the concatenation of its inputs. Each element $$\mathrm{head}_j$$ represents the output derived from applying Query-Key-Value (QKV) attention to a specific sub-space of the initial representation. Finally, the concatenated results are projected via multiplication with a parameter matrix $$\mathbf{W}^{\mathrm{head}} \in \mathbb{R}^{d \times d}$$ to yield the final sequence representation.

Multi-Head Attention Output Calculation

In a multi-head attention mechanism, the model's overall embedding dimension is 768. If this mechanism is configured with 12 separate, parallel attention heads, what is the dimension of the output vector produced by a single one of these heads?

A transformer model has an overall embedding dimension, let's call it `d_model`. Inside this model, a multi-head attention layer is configured with a certain number of parallel attention heads, let's call this number `τ`. Each of these individual heads produces an output vector with its own dimension, `d_h`. Describe the mathematical relationship between `d_model`, `τ`, and `d_h`. Furthermore, explain why this specific relationship is crucial for integrating the multi-head attention layer's final output back into the model's subsequent layers.

Relationship Between Head and Model Dimensions

In a multi-head attention mechanism where the overall model dimension is `d_model` and there are `τ` parallel attention heads (where `τ > 1`), the output vector of a single attention head has a dimension of `d_model`.

Learn Before

Related