1Cademy - Sinusoidal Positional Encoding

Learn Before

Classification of Generalization Approaches for Positional Embeddings

Concept

Sinusoidal Positional Encoding

A simple and dominant scheme for fixed positional encodings relies on sine and cosine functions. Suppose the input representation $\mathbf{X} \in \mathbb{R}^{n imes d}$ contains the $d$ -dimensional embeddings for $n$ sequence tokens. The sinusoidal positional encoding outputs $\mathbf{X} + \mathbf{P}$ using a positional embedding matrix $\mathbf{P} \in \mathbb{R}^{n imes d}$ of the exact same shape, seamlessly adding the positional information directly to the token embeddings.