A language model architect is adjusting the dimensionality d of a model's embeddings, which affects the periods of the sine and cosine functions used for positional information. The period T_k for the k-th component is calculated using the formula: T_k = 2π * b^(2(k-1)/d), where b is a fixed base greater than 1. If the architect increases the dimensionality d while keeping the component index k (where k > 1) and base b constant, how will the period T_k be affected?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model architect is adjusting the dimensionality
dof a model's embeddings, which affects the periods of the sine and cosine functions used for positional information. The periodT_kfor the k-th component is calculated using the formula:T_k = 2π * b^(2(k-1)/d), wherebis a fixed base greater than 1. If the architect increases the dimensionalitydwhile keeping the component indexk(wherek > 1) and basebconstant, how will the periodT_kbe affected?Calculating RoPE Component Period
Diagnosing RoPE Implementation Anomaly