Derivation of the Dot Product for RoPE-Encoded Vectors
The dot product of two RoPE-encoded vectors in 2D Euclidean space, one at position and another at position , can be simplified to demonstrate its dependency on the relative position . The derivation uses the matrix representation of the rotation and properties of the transpose operation: This result shows that the dot product is equivalent to rotating vector by the relative angle and then taking its dot product with vector .
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Derivation of the Dot Product for RoPE-Encoded Vectors
Implicit Relative Position Modeling in Self-Attention with RoPE
A language model uses a positional encoding scheme with a specific mathematical property: the dot product between the encoded representations of any two tokens is a function solely of the difference between their positions in the sequence. Which of the following statements most accurately analyzes the primary advantage of this property for processing language?
In a system that encodes token positions by rotating their vector representations, the dot product between the encoded vector for a token at position
tand another at positionsis found to be dependent only on their relative displacement(t-s). Based on this property, the dot product calculated for a pair of tokens at positions 5 and 8 would be identical to the dot product for the same pair of tokens if they were located at positions 15 and 18.Diagnosing a Positional Encoding Flaw
Learn After
A student is simplifying the dot product between two 2D vectors, x and y, which have been encoded with rotational positional information at positions t and s, respectively. The encoded vectors are given by x' = xRtθ and y' = yRsθ, where Rα is a rotation matrix for angle α. The student's derivation for the dot product x' ⋅ y' is shown below. Identify the step that contains a mathematical error.
Step 1: x' ⋅ y' = (xRtθ)(yRsθ)T Step 2: = xRtθ(Rsθ)TyT Step 3: = xRtθRsθyT Step 4: = xR(t+s)θyT
A key property of certain positional embeddings is that the dot product between two encoded vectors depends on their relative position. The derivation for this property involves several steps. Arrange the following mathematical expressions to show the correct logical sequence for simplifying the dot product of two 2D vectors, x and y, which have been rotated by angles tθ and sθ respectively. The rotated vectors are given by xRtθ and yRsθ.
Justification of RoPE Dot Product Simplification