Formula for the Inner Product of RoPE-Encoded Tokens in Complex Space
The inner product between two RoPE-encoded tokens at positions and , represented by and , is calculated using their complex representations. The formula is: Here, represents the complex form of the original token embedding , and is the complex conjugate of the complex form of embedding . This equation highlights that the relationship between the two tokens is dependent on their relative position, .
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Dot Product of RoPE-Encoded Vectors as a Function of Relative Position
In a model where token positions are encoded by rotating their vector representations, the inner product is calculated between the transformed representations of token 'A' and token 'B'. In Scenario 1, token 'A' is at position 5 and token 'B' is at position 8. In Scenario 2, the same tokens 'A' and 'B' are at positions 12 and 15, respectively. Based on the fundamental property of this encoding method, what is the expected relationship between the inner product value from Scenario 1 and the value from Scenario 2?
Analysis of Rotational Embedding Properties
Formula for the Inner Product of RoPE-Encoded Tokens in Complex Space
Explaining Positional Invariance in Rotational Embeddings
Learn After
Modeling Relative Position Offset via RoPE's Inner Product
The inner product of two token embeddings,
xandy, at positionstandsrespectively, is calculated after a rotational transformation using the formula:⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). In this formula,x'andȳ'are complex number representations of the original embeddings. If both tokens are shifted by a constant amountkto new positionst+kands+k, how does the inner product change?Deconstructing the RoPE Inner Product Formula
The formula for the inner product of two RoPE-encoded tokens is given by
⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). Match each component of this formula to its correct description, analyzing its specific role in the overall calculation.