University of Michigan - Ann Arbor

- Multi-head self-attention: multiple attention projections are computed and then concatenated into a single $D_m$ representation

- Masked attention: self-attention modules in the decoder are adapted to prevent each position from attending to subsequent position

- Cross-attention: in the decoder, the queries are projected from the outputs of the previous (decoder) layer, whereas the keys and values are projected using the outputs of the encoder

Attention in vanilla Transformers

In the encoder, the data will first go through a module called ‘self-attention’ to get a weighted feature x.

Self-Attention of Transformer

Match each vanilla Transformer attention mechanism to its architectural definition.

Describe how the multiple attention projections are combined to produce the final layer representation, and identify the dimension of this resulting representation.

Learn Before

Related