Based on the formula for applying the rotary transformation, $\mathbf{e}_i = \mathrm{Ro}(\mathbf{x}_i, i\theta)$, which input to the transformation function is most likely being ignored or mishandled in the engineer's faulty implementation? Justify your reasoning.

Google

The final embedding for a token at position $i$, denoted as $\mathbf{e}_i$, is obtained by applying the Rotary Positional Embedding (RoPE) transformation to the token's original embedding $\mathbf{x}_i$. This is represented by the function $\mathrm{Ro}(\mathbf{x}_i, i\theta)$, where $i$ is the position and $\theta$ represents the rotational frequency parameters. The formula is: $$\mathbf{e}_i = \mathrm{Ro}(\mathbf{x}_i, i\theta)$$

Application of RoPE to Token Embeddings

The Rotary Positional Embedding (RoPE) rotation is applied to a two-dimensional vector $x = \begin{bmatrix} x_1 & x_2 \end{bmatrix}$ by post-multiplying the vector by the corresponding 2D rotation matrix, $R_\theta$. The resulting rotated vector's components are calculated as follows: $$ Ro(x, \theta) = xR_\theta = \begin{bmatrix} \cos \theta \cdot x_1 - \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix} $$

Application of RoPE Rotation to a 2D Vector

In Rotary Positional Embeddings (RoPE), the rotations applied to different pairs of dimensions are controlled by a set of frequency parameters, denoted by the vector $\theta$. This vector is defined as $\theta = [\theta_1, \dots, \theta_{d/2}]$, where each component $\theta_k$ corresponds to the base frequency for the k-th 2x2 rotation block.

RoPE Frequency Parameters

The 2x2 rotation matrix $R_{t\theta_k}$ is the fundamental building block of the larger block-diagonal matrix used in Rotary Positional Embeddings (RoPE). It rotates a pair of dimensions by an angle $t\theta_k$, where $t$ represents the position and $\theta_k$ is a specific frequency parameter. The matrix is defined as: $$ R_{t\theta_k} = \begin{bmatrix} \cos t\theta_k & \sin t\theta_k \\ -\sin t\theta_k & \cos t\theta_k \end{bmatrix} $$

Definition of the 2x2 RoPE Rotation Matrix Block

In the Rotary Positional Embedding (RoPE) model, the rotational transformation is governed by a set of parameters denoted by the vector $\theta$. This vector consists of $d/2$ components, where $d$ is the dimensionality of the token embeddings. The parameter vector is defined as $\theta = \{\theta_1, \dots, \theta_{d/2}\}$.

RoPE Parameter Vector Definition

In the context of Rotary Positional Embeddings (RoPE), the parameter vector $\theta$ consists of $d/2$ distinct parameters, where $d$ is the dimensionality of the token embeddings. This vector is formally defined as $\theta = (\theta_1, \dots, \theta_{d/2})$. These parameters are used to determine the rotational frequencies applied to the embeddings.

Definition of RoPE Parameter Vector (θ)

A language model encodes token positions by applying a unique, position-dependent rotational transformation to each token's initial embedding. The final, position-aware embedding for a token is the result of this transformation. If the exact same token (e.g., 'model') appears at position 4 and later at position 12 in a sequence, which statement best describes the relationship between their final embeddings, $\mathbf{e}_4$ and $\mathbf{e}_{12}$?

The Rotary Positional Embedding (RoPE) for a 2-dimensional vector $\mathbf{x} = [x_1, x_2]$ is defined as a rotation by an angle $\theta$. This is achieved by multiplying the vector with a 2x2 rotation matrix $R_\theta$. The operation and its result are given by:
$$
\begin{aligned}
\mathrm{Ro}(\mathbf{x}, \theta) &= \mathbf{x}R_\theta \\
&= \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} \cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{bmatrix} \\
&= \begin{bmatrix} \cos \theta \cdot x_1 - \sin \theta \cdot x_2 & \sin \theta \cdot x_1 + \cos \theta \cdot x_2 \end{bmatrix}
\end{aligned}
$$

RoPE 2D Vector Rotation Formula

The final Rotary Positional Embedding (RoPE) for a token embedding `xi` at position `i` is denoted as `ei`. It is calculated by applying the RoPE transformation, `Ro`, to the original embedding `xi` using the position `i` and the set of frequency parameters `θ`. The formula is: $$ e_i = \mathrm{Ro}(\mathbf{x}_i, i\theta) $$

Formula for RoPE-Encoded Token Embedding

A language model generates a final, position-aware embedding, $\mathbf{e}_i$, by applying a rotational transformation to a token's initial embedding, $\mathbf{x}_i$, based on its position, $i$. The process is described by the function $\mathbf{e}_i = \mathrm{Ro}(\mathbf{x}_i, i\theta)$. If two *different* tokens (with distinct initial embeddings $\mathbf{x}_A$ and $\mathbf{x}_B$) are located at the *same* position $p$, is it possible for them to have identical final embeddings (i.e., $\mathbf{e}_A = \mathbf{e}_B$)? Explain your reasoning based on the properties of a rotational transformation.

Learn Before

Related