Origin of NTK-Aware Scaled RoPE

Formula for Scaled RoPE Frequency Parameters (θ')

An engineer extends the context window of a language model that uses rotary positional embeddings. After modification, they find the model struggles with tasks requiring an understanding of long-range dependencies, as if the relative positioning of distant tokens is lost. Which of the following statements best analyzes the fundamental reason for this failure?

Two engineers are modifying a language model's Rotary Positional Embeddings (RoPE) to handle longer text sequences.

• Engineer A proposes modifying the core RoPE transformation function itself (creating a new function, Ro') while keeping the original positional angles (θ) the same.
• Engineer B proposes keeping the original RoPE transformation function (Ro) unchanged but applying it to a new, scaled set of positional angles (θ').

To ensure that the relative positional information is preserved correctly during this context extension, a key condition must be met: the outcome of the new system must be equivalent to the outcome of the original system applied to scaled positions. Based on this principle, which engineer's approach is more theoretically sound, and why?

Interpreting the RoPE Scaling Condition