Calculating Scaled Positional Indices
A language model was trained with a maximum context window of 4096 positions. If this model is now used to process a sequence of 16384 positions using an interpolation technique, explain how the position for the 8192nd token in the new sequence would be re-scaled to fit within the model's original learned range. Describe the calculation and state the resulting scaled position.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Implementing Linear Scaling by Modifying Embedding Model Input
A language model was originally developed to process text sequences with a maximum length of 2048 positions. To enable it to handle a longer input sequence of 8192 positions, a technique is applied that linearly scales down the new position indices to fit within the model's original learned range. Given this scenario, what would be the scaled-down position index that corresponds to the token at position 6144 in the new, longer sequence?
Adapting a Language Model for Longer Documents
Calculating Scaled Positional Indices