Extrapolation of Positional Embeddings
Extrapolation is a positional embedding generalization approach where a model trained on a specific range of observed positions is directly used to assign values to data points beyond that original range. In this method, a function is learned to fit the positional data within the training sequence and is subsequently applied to estimate embeddings for new, unobserved positions outside of it.
0
1
Tags
Foundations of Large Language Models
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Goal of Position Interpolation
A language model was originally trained to understand text sequences with a maximum of 2048 distinct positions. It now needs to process a document that requires 4096 positions. To handle this, a developer implements a technique that rescales the new, larger set of positions (0 to 4095) to fit within the model's original, smaller range (0 to 2047). Which underlying principle does this technique exemplify?
A large language model, trained exclusively on text sequences with a maximum length of 1024 tokens, is later used to process a 3000-token document. The model's positional encoding system simply continues its established pattern to assign unique positions for all tokens up to 3000. Observers note a significant drop in performance, especially in tasks requiring an understanding of relationships between distant parts of the text. Which statement best analyzes this performance issue?
Adapting Positional Embeddings for Longer Contexts
Extrapolation of Positional Embeddings
Example of Positional Extrapolation