Trade-offs in Sequence Architecture Selection
When comparing CNNs, RNNs, and self-attention for sequence tasks, each architecture presents distinct trade-offs. Both CNNs and self-attention support highly parallel computation due to their sequential operations, unlike RNNs. Additionally, self-attention provides the absolute shortest maximum path length of , making it optimal for capturing long-range dependencies. However, because its computational complexity scales quadratically with the sequence length, , self-attention becomes prohibitively slow for very long sequences.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Related
RNN Sequence Processing Complexity
Self-Attention Sequence Processing Complexity
Trade-offs in Sequence Architecture Selection
CNN Sequence Processing Complexity
Match each sequence-to-sequence evaluation parameter or component to its definition.
When comparing architectures for sequence mapping, requiring a smaller number of sequential operations is desirable because it enables ___ computation.
Which architectural evaluation property should the team focus on addressing to resolve this issue, and what specific characteristic should they seek in an alternative architecture?