Definition

Permuted Input Vector in RoPE

In the element-wise formulation of Rotary Positional Embeddings (RoPE), a permuted version of the input vector x\mathbf{x} is required. This vector is constructed by swapping each pair of adjacent elements (x2k1,x2k)(x_{2k-1}, x_{2k}) and negating the first element of the swapped pair, resulting in (x2k,x2k1)(-x_{2k}, x_{2k-1}). For a full d-dimensional vector x\mathbf{x}, the permuted vector is given by: (x2x1xdxd1)\begin{pmatrix} -x_2 \\ x_1 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix} This vector is a key component in the calculation, where it is multiplied element-wise with the sine vector.

Image 0

0

1

Updated 2025-10-08

Contributors are:

Who are from:

Tags

Ch.3 Prompting - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Related