The inner product of two token embeddings, x and y, at positions t and s respectively, is calculated after a rotational transformation using the formula: ⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). In this formula, x' and ȳ' are complex number representations of the original embeddings. If both tokens are shifted by a constant amount k to new positions t+k and s+k, how does the inner product change?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Modeling Relative Position Offset via RoPE's Inner Product
The inner product of two token embeddings,
xandy, at positionstandsrespectively, is calculated after a rotational transformation using the formula:⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). In this formula,x'andȳ'are complex number representations of the original embeddings. If both tokens are shifted by a constant amountkto new positionst+kands+k, how does the inner product change?Deconstructing the RoPE Inner Product Formula
The formula for the inner product of two RoPE-encoded tokens is given by
⟨C(x, tθ), C(y, sθ)⟩ = (x'ȳ')e^(i(t-s)θ). Match each component of this formula to its correct description, analyzing its specific role in the overall calculation.