Learn Before
Explain the sequence length extrapolation capability of fixed sinusoidal positional encodings in the Transformer architecture. In your response, define what this capability entails and explain why it served as the deciding factor in selecting sinusoidal encodings over learned positional embeddings.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Sinusoidal and Learned Positional Encodings - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
When evaluating learned positional embeddings against the base sinusoidal model on the newstest2013 English-to-German development set, what was the empirical outcome?
Explain the sequence length extrapolation capability of fixed sinusoidal positional encodings in the Transformer architecture. In your response, define what this capability entails and explain why it served as the deciding factor in selecting sinusoidal encodings over learned positional embeddings.