Learn Before
Origin of NTK-Aware Scaled RoPE
The NTK-Aware Scaled RoPE method, which is based on the period matching equation, was first introduced in the context of enabling LLaMA models to handle longer sequence lengths.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Related
Formula for Scaled RoPE Frequency Parameters (θ')
An engineer extends the context window of a language model that uses rotary positional embeddings. After modification, they find the model struggles with tasks requiring an understanding of long-range dependencies, as if the relative positioning of distant tokens is lost. Which of the following statements best analyzes the fundamental reason for this failure?
Two engineers are modifying a language model's Rotary Positional Embeddings (RoPE) to handle longer text sequences.
- Engineer A proposes modifying the core RoPE transformation function itself (creating a new function, Ro') while keeping the original positional angles (θ) the same.
- Engineer B proposes keeping the original RoPE transformation function (Ro) unchanged but applying it to a new, scaled set of positional angles (θ').
To ensure that the relative positional information is preserved
Interpreting the RoPE Scaling Condition
Related
Formula for Scaled RoPE Frequency Parameters (θ')
An engineer extends the context window of a language model that uses rotary positional embeddings. After modification, they find the model struggles with tasks requiring an understanding of long-range dependencies, as if the relative positioning of distant tokens is lost. Which of the following statements best analyzes the fundamental reason for this failure?
Two engineers are modifying a language model's Rotary Positional Embeddings (RoPE) to handle longer text sequences.
- Engineer A proposes modifying the core RoPE transformation function itself (creating a new function, Ro') while keeping the original positional angles (θ) the same.
- Engineer B proposes keeping the original RoPE transformation function (Ro) unchanged but applying it to a new, scaled set of positional angles (θ').
To ensure that the relative positional information is preserved
Interpreting the RoPE Scaling Condition