Deconstructing the RoPE Inner Product Formula
Consider the formula for the inner product of two rotationally-encoded token embeddings: ⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). Deconstruct this formula by identifying the component responsible for encoding the tokens' intrinsic content and the component responsible for encoding their relative positions. Explain the reasoning for your identification.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Modeling Relative Position Offset via RoPE's Inner Product
The inner product of two token embeddings,
xandy, at positionstandsrespectively, is calculated after a rotational transformation using the formula:⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). In this formula,x'andȳ'are complex number representations of the original embeddings. If both tokens are shifted by a constant amountkto new positionst+kands+k, how does the inner product change?Deconstructing the RoPE Inner Product Formula
The formula for the inner product of two RoPE-encoded tokens is given by
⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). Match each component of this formula to its correct description, analyzing its specific role in the overall calculation.