Google

To adjust the period of the sine and cosine functions in Rotary Positional Embeddings for longer sequences, one approach is to scale up the period $$T_k$$ by a factor of $$m/m_l$$. The adjusted period is given by: $$T'_k = \frac{m}{m_l} \cdot 2\pi \cdot b^{\frac{2(k-1)}{d}}$$ where $$m$$ is the new sequence length, $$m_l$$ is the original sequence length, and $$b$$ is the base.

Formula for Scaling the Period in Position Interpolation

A language model developer is adapting a model originally trained with a maximum sequence length of 4096 tokens to now work with a maximum sequence length of only 2048 tokens. They use the standard period scaling formula for position interpolation: $T'_k = \frac{m}{m_l} \cdot T_k$, where $m$ is the new sequence length and $m_l$ is the original maximum length. Analyze the effect of this change on the periods of the positional encoding functions. Will the periods increase or decrease, and by what factor? Explain your reasoning based on the formula.

Analyzing Period Scaling Effects

Based on the case study below, analyze the primary consequence of the engineer's error on the model's positional encodings when processing the new, longer sequences.

Diagnosing an Error in Positional Encoding Scaling

A language model was originally trained with a maximum sequence length of 2048 tokens. To handle longer documents, its positional encodings are being adjusted to accommodate a new sequence length of 8192 tokens by scaling the periods of the encoding functions. If the original period for a specific dimension was 100, what is the new, adjusted period for that same dimension after this adjustment?

Learn Before

Related