1Cademy - General Equivalence of Modified RoPE

Formula

General Equivalence of Modified RoPE

A modified Rotary Positional Embedding function, denoted Ro', can be defined through its equivalence to the original Ro function. Applying the modified function Ro' to a token embedding $\mathbf{x}_i$ with position parameters $i\theta$ is identical to applying the original function Ro to the same embedding but with a transformed set of position parameters, $i\theta'$ . This relationship is formally stated as: $\text{Ro}'(\mathbf{x}_i, i\theta) = \text{Ro}(\mathbf{x}_i, i\theta')$

Updated 2026-05-02

Contributors are:

Who are from:

References

Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course

Learn After

Formula for RoPE with Linear Positional Interpolation
A researcher defines a new rotary position embedding function, Ro_new, for a token x_i at position i. The new function is defined as Ro_new(x_i, iθ) = Ro(x_i, (i+c)θ), where Ro is the original function and c is a constant offset. According to the general equivalence principle, this can be written as Ro_new(x_i, iθ) = Ro(x_i, iθ'). What is the correct expression for the transformed position parameter iθ'?
RoPE Scaling Transformation Equivalence
Equivalence of RoPE Modification Strategies
Analysis of a Flawed RoPE Modification