Google

The formula $$\mathbf{k}'_i = \phi(\mathbf{k}_i)$$ describes the transformation of an original vector $$\mathbf{k}_i$$ into a new vector $$\mathbf{k}'_i$$ by applying a transformation function or map $$\phi$$. In this formula, the bold notation indicates vector quantities, and the subscript $$i$$ denotes that the operation is performed on an individual element within a set or sequence of vectors. In the context of Large Language Models, this formula can represent the transformation of key vectors within an attention mechanism.

Vector Transformation Formula

The Rotary Position Embedding (RoPE) transformation can be represented in d-dimensional Euclidean space using the formula: $$ \mathrm{Ro}(\mathbf{x}, t\theta) = \begin{bmatrix} x_1 & x_2 & \dots & x_d \end{bmatrix} \begin{bmatrix} R_{t\theta_1} & & & \\ & R_{t\theta_2} & & \\ & & \ddots & \\ & & & R_{t\theta_{d/2}} \end{bmatrix} $$ In this equation, a d-dimensional vector $$\mathbf{x}$$ is transformed by multiplying it with a block-diagonal matrix. This matrix consists of $$d/2$$ individual 2x2 rotation matrices, $$R_{t\theta_k}$$, along its diagonal. This structure effectively pairs up the components of $$\mathbf{x}$$ (e.g., $$x_1$$ with $$x_2$$, $$x_3$$ with $$x_4$$, etc.) and applies a separate 2D rotation to each pair.

Euclidean Representation of RoPE Transformation

In the context of the vector transformation formula $\mathbf{k}'_i = \phi(\mathbf{k}_i)$, analyze the following scenario and identify the element that corresponds to each component of the formula: $\mathbf{k}_i$, $\phi$, and $\mathbf{k}'_i$.

Identifying Components of a Vector Transformation

Match each component of the vector transformation formula $\mathbf{k}'_i = \phi(\mathbf{k}_i)$ with its correct description.

In $$\mathbf{k}'_i = \phi(\mathbf{k}_i)$$, the function $$\phi$$ maps the indexed input vector $$\mathbf{k}_i$$ to the transformed vector $$\mathbf{k}'_i$$. Bold symbols denote vectors, the subscript $$i$$ identifies a vector within a sequence, and the prime distinguishes the output from the input. The expression does not specify how $$\phi$$ performs the transformation, so it does not by itself imply a matrix-vector product or any other particular operation.

Interpretation of Indexed Vector-Transformation Notation

Consider an original 2-dimensional vector represented as $\mathbf{k} = [4, 7]$. If a transformation function, denoted by $\phi$, is defined as multiplying the first element of an input vector by 2 and adding 1 to the second element, what is the resulting transformed vector $\mathbf{k}'$ after applying the formula $\mathbf{k}' = \phi(\mathbf{k})$?

Learn Before

Related