Formula for the Period of RoPE's Sine and Cosine Components
The period, , of the sine and cosine functions used in Rotary Positional Embeddings (RoPE) is determined by the frequency parameter . The formula for the period is given by: where is the base, is the component index, and is the embedding dimensionality. This formula is derived from the relationship .
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Related
Formula for the Period of RoPE's Sine and Cosine Components
In a Rotary Positional Embedding system, a specific component
kuses sine and cosine functions of the positionito create a rotational transformation. For this component, these functions have a period ofT. How does the rotational transformation for this component at positionicompare to the rotational transformation at positioni + T?Analyzing the Role of Period Length in RoPE
Analyzing Positional Encoding in RoPE
Calculation of RoPE Frequency Parameters
Formula for the Period of RoPE's Sine and Cosine Components
Consider the generalized formula for calculating a set of frequency parameters: In this formula,
bis a configurable base greater than 1,dis the dimensionality (a positive integer), andkis the component index, which is an integer greater than 1. How would increasing the value of the basebaffect the calculated frequencyθ_kfor any givenkandd?Determining the Base from a Frequency Parameter
Tuning Positional Embeddings for Long-Context Models
Learn After
A language model architect is adjusting the dimensionality
dof a model's embeddings, which affects the periods of the sine and cosine functions used for positional information. The periodT_kfor the k-th component is calculated using the formula:T_k = 2π * b^(2(k-1)/d), wherebis a fixed base greater than 1. If the architect increases the dimensionalitydwhile keeping the component indexk(wherek > 1) and basebconstant, how will the periodT_kbe affected?Calculating RoPE Component Period
Diagnosing RoPE Implementation Anomaly