In a model where token positions are encoded by rotating their vector representations, the inner product is calculated between the transformed representations of token 'A' and token 'B'. In Scenario 1, token 'A' is at position 5 and token 'B' is at position 8. In Scenario 2, the same tokens 'A' and 'B' are at positions 12 and 15, respectively. Based on the fundamental property of this encoding method, what is the expected relationship between the inner product value from Scenario 1 and the value from Scenario 2?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Dot Product of RoPE-Encoded Vectors as a Function of Relative Position
In a model where token positions are encoded by rotating their vector representations, the inner product is calculated between the transformed representations of token 'A' and token 'B'. In Scenario 1, token 'A' is at position 5 and token 'B' is at position 8. In Scenario 2, the same tokens 'A' and 'B' are at positions 12 and 15, respectively. Based on the fundamental property of this encoding method, what is the expected relationship between the inner product value from Scenario 1 and the value from Scenario 2?
Analysis of Rotational Embedding Properties
Formula for the Inner Product of RoPE-Encoded Tokens in Complex Space
Explaining Positional Invariance in Rotational Embeddings