Learn Before
Formula for RoPE Frequency Parameters (θ_k)
The frequency parameter in Rotary Positional Embeddings (RoPE) is defined as an exponential function of its index . The specific formula is: where is a predefined base, is the index for each pair of dimensions (ranging from to ), and is the total dimensionality of the embedding. This formulation creates a geometric progression of frequencies.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Related
In a system that uses rotational transformations to encode positional information, the frequencies of rotation for different pairs of dimensions (θ_k) are defined as an exponential function of their index (k). What is the primary analytical consequence of this design choice for the resulting positional encodings?
Evaluating Positional Encoding Schemes
Formula for RoPE Frequency Parameters (θ_k)
In a system that uses rotational transformations to encode positional information, where the rotation frequencies are defined as an exponential function of their dimension index, the frequencies of rotation are designed to decrease for higher-indexed pairs of dimensions.
Learn After
Calculation of RoPE Frequency Parameters
Formula for the Period of RoPE's Sine and Cosine Components
Consider the generalized formula for calculating a set of frequency parameters: In this formula,
bis a configurable base greater than 1,dis the dimensionality (a positive integer), andkis the component index, which is an integer greater than 1. How would increasing the value of the basebaffect the calculated frequencyθ_kfor any givenkandd?Determining the Base from a Frequency Parameter
Tuning Positional Embeddings for Long-Context Models