Learn Before
Compare Multi-Head Attention to a single attention head in terms of representation capability, and explain how the dimensions chosen for individual heads affect total computational cost.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Multi-Head Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
In the Multi-Head Attention mechanism, what are the dimensions of the final output projection parameter matrix ?
In Multi-Head Attention, each of the parallel attention heads shares an identical set of projection parameter matrices , , and .
What operation is applied to the individual parallel head outputs () immediately before multiplying them by the parameter matrix ?
Compare Multi-Head Attention to a single attention head in terms of representation capability, and explain how the dimensions chosen for individual heads affect total computational cost.