Learn Before
Origin of NTK-Aware Scaled RoPE
The NTK-Aware Scaled RoPE method, which is based on the period matching equation, was first introduced in the context of enabling LLaMA models to handle longer sequence lengths.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Related
Origin of NTK-Aware Scaled RoPE
Formula for Scaled RoPE Frequency Parameters (θ')
An engineer extends the context window of a language model that uses rotary positional embeddings. After modification, they find the model struggles with tasks requiring an understanding of long-range dependencies, as if the relative positioning of distant tokens is lost. Which of the following statements best analyzes the fundamental reason for this failure?
Two engineers are modifying a language model's Rotary Positional Embeddings (RoPE) to handle longer text sequences.
- Engineer A proposes modifying the core RoPE transformation function itself (creating a new function, Ro') while keeping the original positional angles (θ) the same.
- Engineer B proposes keeping the original RoPE transformation function (Ro) unchanged but applying it to a new, scaled set of positional angles (θ').
To ensure that the relative positional information is preserved
Interpreting the RoPE Scaling Condition