Explaining Positional Invariance in Rotational Embeddings
In a system that uses rotational transformations to encode token positions, an engineer observes that the inner product between the representations of the words 'the' (at position 3) and 'cat' (at position 5) is identical to the inner product between the same words when they appear at positions 10 and 12. Briefly explain the mathematical property of this encoding scheme that accounts for this observation.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Dot Product of RoPE-Encoded Vectors as a Function of Relative Position
In a model where token positions are encoded by rotating their vector representations, the inner product is calculated between the transformed representations of token 'A' and token 'B'. In Scenario 1, token 'A' is at position 5 and token 'B' is at position 8. In Scenario 2, the same tokens 'A' and 'B' are at positions 12 and 15, respectively. Based on the fundamental property of this encoding method, what is the expected relationship between the inner product value from Scenario 1 and the value from Scenario 2?
Analysis of Rotational Embedding Properties
Formula for the Inner Product of RoPE-Encoded Tokens in Complex Space
Explaining Positional Invariance in Rotational Embeddings