Comparing CNN, RNN, and Self-Attention Architectures
When evaluating architectures such as CNNs, RNNs, and self-attention for mapping an input sequence of tokens to an output sequence of the same length (with each token represented as a -dimensional vector), three main properties are compared: computational complexity, sequential operations, and maximum path lengths. A smaller number of sequential operations is desirable as it allows for parallel computation, while a shorter maximum path length between tokens makes it easier for the network to learn long-range dependencies.
0
1
References
Tags
D2L
Dive into Deep Learning @ D2L
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Complexity and Path Lengths in Self-Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
RNN Sequence Processing Complexity
Self-Attention Sequence Processing Complexity
Trade-offs in Sequence Architecture Selection
CNN Sequence Processing Complexity
Match each sequence-to-sequence evaluation parameter or component to its definition.
When comparing architectures for sequence mapping, requiring a smaller number of sequential operations is desirable because it enables ___ computation.
Which architectural evaluation property should the team focus on addressing to resolve this issue, and what specific characteristic should they seek in an alternative architecture?
Learn After
RNN Sequence Processing Complexity
Self-Attention Sequence Processing Complexity
Trade-offs in Sequence Architecture Selection
CNN Sequence Processing Complexity
Match each sequence-to-sequence evaluation parameter or component to its definition.
When comparing architectures for sequence mapping, requiring a smaller number of sequential operations is desirable because it enables ___ computation.
Which architectural evaluation property should the team focus on addressing to resolve this issue, and what specific characteristic should they seek in an alternative architecture?