Derivation of the Dot Product for RoPE-Encoded Vectors

Implicit Relative Position Modeling in Self-Attention with RoPE

A language model uses a positional encoding scheme with a specific mathematical property: the dot product between the encoded representations of any two tokens is a function solely of the difference between their positions in the sequence. Which of the following statements most accurately analyzes the primary advantage of this property for processing language?

In a system that encodes token positions by rotating their vector representations, the dot product between the encoded vector for a token at position t and another at position s is found to be dependent only on their relative displacement (t-s). Based on this property, the dot product calculated for a pair of tokens at positions 5 and 8 would be identical to the dot product for the same pair of tokens if they were located at positions 15 and 18.

Diagnosing a Positional Encoding Flaw