Explaining Extrapolation Failure in Positional Embeddings
A language model is designed to understand the order of words by learning a unique numerical representation for each position in a sequence (e.g., position 1, position 2, etc.). This model is trained exclusively on documents that are 512 words long or shorter. Explain the fundamental reason why this model would likely struggle to process a document that is 800 words long.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Classification of Generalization Approaches for Positional Embeddings
Positional Encoding without Generalization
A team trains a language model using an architecture where a unique vector is learned for every possible token position. The entire training dataset consists of texts that are no longer than 1,024 tokens. After training, the model shows excellent performance on all evaluation texts that are 1,024 tokens or shorter. However, when deployed to process a new, 1,500-token document, the model's ability to understand relationships between words degrades dramatically, particularly for words appearing after the 1,024th position. Which of the following is the most direct cause of this performance drop?
Explaining Extrapolation Failure in Positional Embeddings
Evaluating a Flawed Generalization Strategy
Generalizable Positional Embeddings