Element-wise Formula for RoPE Rotation

In a system that encodes positional information, a 'cosine vector' is defined for each position $i$ based on a set of frequency parameters $\theta = [\theta_1, \dots, \theta_{d/2}]$. The formula for this vector is: $[\cos i\theta_1, \dots, \cos i\theta_{d/2}]$.

Given a total dimension $d=4$, a position $i=2$, and frequency parameters $\theta_1 = 0.01$ and $\theta_2 = 0.0001$, what is the correct cosine vector for this position?

A developer is implementing a positional encoding scheme where a 'cosine vector' is needed for each position $i$. Given a set of frequency parameters $\theta = [\theta_1, \dots, \theta_{d/2}]$ and a total embedding dimension $d$, which of the following formulas correctly defines this vector?

Properties of Positional Cosine Vectors

Definition of the RoPE Sine Vector

Periodicity of RoPE's Sine and Cosine Components

The application of a rotary positional embedding to a two-dimensional vector $(x_1, x_2)$ at position $i$ with frequency $\theta$ results in a new vector $(y_1, y_2)$, where the components are calculated as:

$y_1 = x_1 \cos(i\theta) - x_2 \sin(i\theta)$ $y_2 = x_1 \sin(i\theta) + x_2 \cos(i\theta)$

Based on this structure, which statement best analyzes how the output vector's components are formed?

A positional encoding method transforms a two-dimensional vector $(x_1, x_2)$ at position $i$ into a new vector $(y_1, y_2)$ using the equations:

$y_1 = x_1 \cos(i\theta) - x_2 \sin(i\theta)$ $y_2 = x_1 \sin(i\theta) + x_2 \cos(i\theta)$

This transformation is considered non-linear with respect to the input vector $(x_1, x_2)$ because it involves trigonometric functions.

Matrix Representation of a 2D Rotary Transformation