Definition of the 2x2 RoPE Rotation Matrix Block

Calculation of RoPE Frequency Parameters

Exponential Form of RoPE Frequency Parameters

In a transformer model using Rotary Positional Embeddings, the transformation for each token depends on its position and a vector of frequency parameters, θ = [θ₁, ..., θ_{d/2}], where each component θ_k corresponds to a different 2-dimensional rotation. A researcher proposes a modification where all components of this vector are set to the same value (i.e., θ₁ = θ₂ = ... = θ_{d/2}). What is the most likely consequence of this change on the model's ability to represent positional information?

Effect of Modifying RoPE Frequency Parameters

In the implementation of Rotary Positional Embeddings (RoPE), the vector of frequency parameters, θ = [θ₁, ..., θ_{d/2}], ensures that for any given token position, all pairs of dimensions within its embedding are rotated by the same amount.

Application of RoPE Rotation to a 2D Vector

RoPE Frequency Parameters

Definition of the 2x2 RoPE Rotation Matrix Block

RoPE Parameter Vector Definition

Definition of RoPE Parameter Vector (θ)

A language model encodes token positions by applying a unique, position-dependent rotational transformation to each token's initial embedding. The final, position-aware embedding for a token is the result of this transformation. If the exact same token (e.g., 'model') appears at position 4 and later at position 12 in a sequence, which statement best describes the relationship between their final embeddings, $\mathbf{e}_4$ and $\mathbf{e}_{12}$?

RoPE 2D Vector Rotation Formula

Formula for RoPE-Encoded Token Embedding

Uniqueness of RoPE-based Embeddings

Debugging a RoPE Implementation

Application of RoPE Rotation to a 2D Vector

Definition of the 2x2 RoPE Rotation Matrix Block

A developer is implementing a function to rotate a 2D row vector v counter-clockwise by an angle θ. The operation is performed by post-multiplying the vector by a 2x2 matrix R (i.e., v_rotated = vR). Which of the following matrices correctly represents this transformation?

Constructing a 90-Degree Rotation Matrix

Consider a transformation applied to a 2D row vector v by post-multiplying it with a matrix R (i.e., v_rotated = vR). The matrix $R = \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}$ correctly performs a counter-clockwise rotation of the vector by an angle θ.