Google

The inner product between two RoPE-encoded tokens at positions $t$ and $s$ is a function of the term $t - s$. This term directly corresponds to the relative offset between the tokens, demonstrating how RoPE's formulation inherently captures relative positional information.

Modeling Relative Position Offset via RoPE's Inner Product

Consider two token embeddings, **x** and **y**, encoded with a rotational position scheme at positions 10 and 7, respectively. Their resulting inner product is calculated. If these same two tokens are instead placed at positions 25 and 22, how would the new inner product compare to the original one?

A language model uses a rotational scheme to encode token positions. An analyst observes that the inner product between the encoded representations of the word 'apple' at position 4 and 'banana' at position 9 is identical to the inner product between the encoded representations of 'apple' at position 21 and 'banana' at position 26. Explain the mathematical reason for this observation.

Invariance in Rotational Position Encodings

An engineer is analyzing a language model's behavior. They observe that the computed relationship between the token for 'deep' and the token for 'learning' is different when processing the phrase 'a course on deep learning' compared to when processing the phrase 'advances in deep learning for science'. In both phrases, the word 'learning' immediately follows 'deep'.

Based on this observation, evaluate whether the model's positional encoding scheme is correctly implementing the principle that the inner product between two rotationally-encoded tokens should depend solely on their relative offset. Justify your conclusion.

Learn Before

Related