Learn Before
Position Interpolation Mapping for Longer Sequences
Position interpolation maps the positions in a new, longer sequence to match the original position range observed during training. If the training sequence lengths ranged from to , and the new sequence has a length , position interpolation compresses all points in the expanded range into the original learned range .
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Related
Position Interpolation Mapping for Longer Sequences
Period Adjustment in Position Interpolation
Position Interpolation by Scaling the RoPE Base
A large language model was trained exclusively on documents with a maximum length of 2048 tokens. An engineer now needs to use this pre-trained model to process a new document that is 4096 tokens long without altering the model's architecture or retraining it. If the engineer applies a position interpolation technique, what is the fundamental objective of this action?
Analyzing Performance Degradation with Long Sequences
Evaluating a Strategy for Extending Context Length
Example of Interpolation by Scaling Positions
Learn After
Implementing Linear Scaling by Modifying Embedding Model Input
A language model was originally developed to process text sequences with a maximum length of 2048 positions. To enable it to handle a longer input sequence of 8192 positions, a technique is applied that linearly scales down the new position indices to fit within the model's original learned range. Given this scenario, what would be the scaled-down position index that corresponds to the token at position 6144 in the new, longer sequence?
Adapting a Language Model for Longer Documents
Calculating Scaled Positional Indices