Comparing CNN, RNN, and Self-Attention Architectures
When evaluating architectures such as CNNs, RNNs, and self-attention for mapping an input sequence of tokens to an output sequence of the same length (with each token represented as a -dimensional vector), three main properties are compared: computational complexity, sequential operations, and maximum path lengths. A smaller number of sequential operations is desirable as it allows for parallel computation, while a shorter maximum path length between tokens makes it easier for the network to learn long-range dependencies.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Complexity and Path Lengths in Self-Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
RNN Sequence Processing Complexity
Self-Attention Sequence Processing Complexity
Trade-offs in Sequence Architecture Selection
CNN Sequence Processing Complexity
Match each sequence-to-sequence evaluation parameter or component to its definition.
When comparing architectures for sequence mapping, requiring a smaller number of sequential operations is desirable because it enables ___ computation.
Which architectural evaluation property should the team focus on addressing to resolve this issue, and what specific characteristic should they seek in an alternative architecture?