Origin of NTK-Aware Scaled RoPE

Formula for Scaled RoPE Frequency Parameters (θ')

An engineer extends the context window of a language model that uses rotary positional embeddings. After modification, they find the model struggles with tasks requiring an understanding of long-range dependencies, as if the relative positioning of distant tokens is lost. Which of the following statements best analyzes the fundamental reason for this failure?

Two engineers are modifying a language model's Rotary Positional Embeddings (RoPE) to handle longer text sequences.

• Engineer A proposes modifying the core RoPE transformation function itself (creating a new function, Ro') while keeping the original positional angles (θ) the same.
• Engineer B proposes keeping the original RoPE transformation function (Ro) unchanged but applying it to a new, scaled set of positional angles (θ').

To ensure that the relative positional information is preserved

Interpreting the RoPE Scaling Condition