Sinusoidal Positional Encoding
A simple and dominant scheme for fixed positional encodings relies on sine and cosine functions. Suppose the input representation contains the -dimensional embeddings for sequence tokens. The sinusoidal positional encoding outputs using a positional embedding matrix of the exact same shape, seamlessly adding the positional information directly to the token embeddings.
0
1
References
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Dive into Deep Learning
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
D2L
Dive into Deep Learning @ D2L
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Sinusoidal and Learned Positional Encodings - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Sinusoidal Positional Encoding
Extrapolation and Interpolation Methods for Positional Embeddings
Example of Extrapolation in Sequence Models
Comparison of Generalizing vs. Non-Generalizing Positional Encodings
Example of Interpolation in Sequence Models
A language model was trained exclusively on text sequences with a maximum length of 1024 tokens. When presented with a 2048-token sequence, two different approaches are considered for generating positional information for the new, unseen positions (1024 to 2047).
Approach X: The mechanism generates values for the new positions by continuing the mathematical pattern it learned from the original 0-1023 positions.
Approach Y: The mechanism rescales the positional indices of the entire 204
Choosing a Positional Embedding Generalization Strategy
A language model is trained on sequences up to a maximum length of
L. During inference, it encounters a sequence of length2L. Match each strategy for handling the unseen positions (Lto2L-1) with its corresponding classification.Sinusoidal versus Learned Positional Embeddings
Positional Encoding
Sinusoidal Positional Encoding
Learn After
A development team is building a language model that will be trained on documents with a maximum length of 512 tokens. However, a critical requirement for the final application is that the model must effectively process documents that are occasionally up to 4000 tokens long. The team chooses to use a position representation method based on a combination of sine and cosine functions of different frequencies. Which of the following statements most accurately evaluates this choice?
Analyzing the Trade-offs of Sinusoidal Positional Encoding
Match each mathematical component of the sinusoidal positional encoding scheme with its description.
Which two mathematical functions serve as the foundation for the fixed positional encoding scheme described?