Learn Before
Formula for Multi-dimensional RoPE in Complex Space
In a complex space, the rotary positional embedding for a -dimensional token vector can be formulated as: where represents the -th component of the token embedding reinterpreted as a complex number (formed by pairs of adjacent elements), is the position, is the rotation frequency parameter for the -th dimension, and is the standard basis vector with a single non-zero value in the -th coordinate and 0's elsewhere.
0
1
Tags
Foundations of Large Language Models
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Definition of RoPE-Encoded Token Embedding Notation (ei)
A 6-dimensional token embedding vector, represented as
v = [v1, v2, v3, v4, v5, v6], is being prepared for a rotational transformation to encode its position. Which of the following correctly describes how this vector is reinterpreted for the transformation process?Vector Reinterpretation for Rotational Transformation
To prepare a 4-dimensional token embedding vector
v = [1.0, 2.5, -0.5, 4.0]for a rotational transformation, it is first reinterpreted as a 2-dimensional vector of complex numbers. The first complex number in this new vector is1.0 + 2.5i. The second complex number is ____.When applying a rotational transformation to a 128-dimensional token embedding, the transformation applied to the complex number formed by the 1st and 2nd elements is dependent on the transformation applied to the complex number formed by the 3rd and 4th elements.
Formula for Multi-dimensional RoPE in Complex Space