Evaluating RoPE Scaling for Context Extension
Analyze the following scenario and determine the appropriate adjustment.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model's positional embeddings are being adapted for a new context length. The adaptation uses a scaling factor
位 = 0.25, a base termb = 10000, and an embedding dimensiond = 128. Based on the formula for the scaled frequency parameters,胃' = [ (位b)^(-0/d), (位b)^(-2/d), ..., (位b)^(-(d-2)/d) ], what is the approximate value of the second component in the胃'vector?Impact of Scaling Factor on RoPE Frequencies
Related
A language model's positional embeddings are being adapted for a new context length. The adaptation uses a scaling factor
位 = 0.25, a base termb = 10000, and an embedding dimensiond = 128. Based on the formula for the scaled frequency parameters,胃' = [ (位b)^(-0/d), (位b)^(-2/d), ..., (位b)^(-(d-2)/d) ], what is the approximate value of the second component in the胃'vector?Impact of Scaling Factor on RoPE Frequencies