Learn Before
To enable a language model to process sequences longer than its original training limit, the period of its positional encoding functions must be reduced. This adjustment ensures that the new, more distant positions are mapped within the range the model has already learned.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for Scaling the Period in Position Interpolation
A language model was originally trained to handle text up to a maximum of 4096 tokens. To enable it to process a document with 8192 tokens without retraining, a modification is made to its positional encoding functions. Based on the principles of position interpolation, which statement best describes the nature and effect of this modification?
Analyzing a Positional Encoding Modification
Mechanism of Position Interpolation
To enable a language model to process sequences longer than its original training limit, the period of its positional encoding functions must be reduced. This adjustment ensures that the new, more distant positions are mapped within the range the model has already learned.