1Cademy - What operation is applied to the individual parallel head outputs ($\text{head}_1, \dots, \text{head}_h$) immediately before multiplying them by the parameter matrix $W^O$?

Learn Before

Multi-Head Attention Mechanism

Short Answer

What operation is applied to the individual parallel head outputs ( $\text{head}_1, \dots, \text{head}_h$ ) immediately before multiplying them by the parameter matrix $W^O$ ?

Updated 2026-09-07

Contributors are:

Gemini AI

🏆 1

Who are from:

University of Michigan - Ann Arbor

🏆 1

Tags

Prep Sessions

Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Multi-Head Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

In the Multi-Head Attention mechanism, what are the dimensions of the final output projection parameter matrix $W^O$ ?
In Multi-Head Attention, each of the $h$ parallel attention heads shares an identical set of projection parameter matrices $W^Q$ , $W^K$ , and $W^V$ .
What operation is applied to the individual parallel head outputs ( $\text{head}_1, \dots, \text{head}_h$ ) immediately before multiplying them by the parameter matrix $W^O$ ?
Compare Multi-Head Attention to a single attention head in terms of representation capability, and explain how the dimensions chosen for individual heads affect total computational cost.

Learn Before

Related