Inner Product of RoPE-Encoded Token Representations
To analyze how Rotary Positional Embeddings (RoPE) encode relative positional information, the inner product is calculated between the rotated representations of tokens at different positions, such as and . This operation is fundamental to demonstrating that the relationship between two tokens in RoPE depends only on their relative positions.

0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Inner Product of RoPE-Encoded Token Representations
In the context of positional embeddings, the notation represents the result of rotating a 2-dimensional vector by an angle . If the initial vector is and the rotation angle is , what is the resulting vector?
The notation is a shorthand for rotating a 2-dimensional vector by an angle . This operation is equivalent to representing the vector as a complex number and multiplying it by . Based on this, which expression correctly represents the first component of the resulting rotated vector?
The transformation represented by the notation , which rotates a 2-dimensional vector by an angle , changes the magnitude (or length) of the original vector .
Learn After
Dot Product of RoPE-Encoded Vectors as a Function of Relative Position
In a model where token positions are encoded by rotating their vector representations, the inner product is calculated between the transformed representations of token 'A' and token 'B'. In Scenario 1, token 'A' is at position 5 and token 'B' is at position 8. In Scenario 2, the same tokens 'A' and 'B' are at positions 12 and 15, respectively. Based on the fundamental property of this encoding method, what is the expected relationship between the inner product value from Scenario 1 and the value from Scenario 2?
Analysis of Rotational Embedding Properties
Formula for the Inner Product of RoPE-Encoded Tokens in Complex Space
Explaining Positional Invariance in Rotational Embeddings