Instead of executing a single attention function using queries, keys, and values of the model dimension $d_{\text{model}}$, Multi-Head Attention linearly projects $Q$, $K$, and $V$ with $h$ separate learned projections to dimensions $d_k$, $d_k$, and $d_v$, respectively. Applying attention across each of these projected versions in parallel produces $d_v$-dimensional output values that are concatenated and projected once again to form the final values:

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$

$\text{where } \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$

The projections are parameter matrices $W_i^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$, and $W^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}$. In the base model, $h = 8$ parallel attention heads are used with $d_k = d_v = d_{\text{model}} / h = 64$ for $d_{\text{model}} = 512$.

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions simultaneously. In contrast, a single attention head averages attention-weighted positions, which inhibits this capability. Because the dimensions of each head are reduced, the total computational cost remains similar to that of full-dimensionality single-head attention.