Learn Before
Mechanism of Position Interpolation
A language model trained on sequences up to a certain length needs to process a much longer sequence. To achieve this, a technique is used that modifies its positional encoding functions. Explain the specific change made to these functions and why this change enables the model to handle the longer input.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Comprehension in Revised Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for Scaling the Period in Position Interpolation
A language model was originally trained to handle text up to a maximum of 4096 tokens. To enable it to process a document with 8192 tokens without retraining, a modification is made to its positional encoding functions. Based on the principles of position interpolation, which statement best describes the nature and effect of this modification?
Analyzing a Positional Encoding Modification
Mechanism of Position Interpolation
To enable a language model to process sequences longer than its original training limit, the period of its positional encoding functions must be reduced. This adjustment ensures that the new, more distant positions are mapped within the range the model has already learned.