Using the formula for a positionally-encoded token embedding, $$ e_i = \mathrm{Ro}(\mathbf{x}_i, i\theta) $$, explain why this observation is the expected behavior and not a bug.

Google

The final Rotary Positional Embedding (RoPE) for a token embedding `xi` at position `i` is denoted as `ei`. It is calculated by applying the RoPE transformation, `Ro`, to the original embedding `xi` using the position `i` and the set of frequency parameters `θ`. The formula is: $$ e_i = \mathrm{Ro}(\mathbf{x}_i, i\theta) $$

Formula for RoPE-Encoded Token Embedding

In the formula for a positionally-encoded token embedding, `e_i = Ro(x_i, iθ)`, what is the primary function of the `iθ` term?

Analyzing Language Model Embeddings

In the formula for a positionally-encoded token embedding, $$ e_i = \mathrm{Ro}(\mathbf{x}_i, i\theta) $$, analyze the distinct contributions of the $\mathbf{x}_i$ term and the $i\theta$ term to the final embedding $\mathbf{e}_i$.

Learn Before

Related