A language model, originally trained with a maximum sequence length of 2048, is being adapted to handle a new maximum length of 8192 using linear positional interpolation. This technique modifies the rotary position embedding function (Ro) by scaling the position index. The new function, Ro', for a token x_i at position i in the longer sequence, is equivalent to applying the original function Ro with a scaled positional argument: `Ro'(x_i, iθ) = Ro(x_i, (original_max_length / new_max_l

Adapting a Language Model for Longer Sequences

A language model's context window is being extended from an original maximum length m_l to a new, larger maximum length m. The technique used modifies the rotary position embedding function (Ro) by scaling the position index i according to the formula: New Effective Position = (m_l / m) * i. This formula implies that the effective position for a token at the very end of the new, extended context (position m-1) is mapped to a position that falls outside the range of the original model