Learn Before
In a certain positional encoding method, the rotation of a pair of dimensions is achieved using the matrix , where t is the position and \theta is a frequency parameter.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.3 Prompting - Foundations of Large Language Models
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Euclidean Representation of RoPE Rotation
In a specific positional encoding method, pairs of embedding dimensions are rotated using a 2x2 matrix defined as: where
tis the token's position and\theta_kis a frequency. Given a token at positiont = 2and a frequency\theta_k = \pi/4, which matrix correctly represents the rotation?In a certain positional encoding method, the rotation of a pair of dimensions is achieved using the matrix , where
tis the position and\thetais a frequency parameter.Applying a Rotational Transformation Matrix