Learn Before
Based on path length analysis, explain how adopting restricted self-attention changes the maximum path length compared to full self-attention, and describe the resulting effect on gradient flow and long-range dependency learning.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Complexity and Path Lengths in Self-Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Match each layer architecture to its corresponding maximum path length between arbitrary input and output positions.
Connecting all pairs of input and output positions using a stack of convolutional layers with contiguous kernels of width k requires a maximum path length of ___.
Order these layer architectures from shortest to longest maximum path length between arbitrary input and output positions (assuming kernel width 1 < k < n):
Based on path length analysis, explain how adopting restricted self-attention changes the maximum path length compared to full self-attention, and describe the resulting effect on gradient flow and long-range dependency learning.