Learn Before
Analysis of a Flawed RoPE Modification
A key principle for modifying rotary positional embeddings states that the effect of a new function, Ro', can be understood as applying the original function, Ro, with a transformed position parameter, iθ'. This is expressed as: Based on this principle, analyze the following scenario.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for RoPE with Linear Positional Interpolation
A researcher defines a new rotary position embedding function,
Ro_new, for a tokenx_iat positioni. The new function is defined asRo_new(x_i, iθ) = Ro(x_i, (i+c)θ), whereRois the original function andcis a constant offset. According to the general equivalence principle, this can be written asRo_new(x_i, iθ) = Ro(x_i, iθ'). What is the correct expression for the transformed position parameteriθ'?RoPE Scaling Transformation Equivalence
Equivalence of RoPE Modification Strategies
Analysis of a Flawed RoPE Modification