Learn Before
In self-attention layers, computing representations requires waiting for earlier sequence positions to complete before later positions can be processed.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Complexity and Path Lengths in Self-Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
In natural language processing tasks using word-piece or byte-pair representations, why are self-attention layers typically computationally faster than recurrent layers?
In self-attention layers, computing representations requires waiting for earlier sequence positions to complete before later positions can be processed.
Why do standard recurrent layers require sequential operations to process a sequence of length ?
Explain the motivation, mechanism, and resulting computational profile (both per-layer complexity and sequential operations) of restricted self-attention for long sequences.