Diagnosing a Positional Encoding Flaw
Based on the case study below, identify the most likely flaw in the model's positional encoding scheme and explain your reasoning.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Derivation of the Dot Product for RoPE-Encoded Vectors
Implicit Relative Position Modeling in Self-Attention with RoPE
A language model uses a positional encoding scheme with a specific mathematical property: the dot product between the encoded representations of any two tokens is a function solely of the difference between their positions in the sequence. Which of the following statements most accurately analyzes the primary advantage of this property for processing language?
In a system that encodes token positions by rotating their vector representations, the dot product between the encoded vector for a token at position
tand another at positionsis found to be dependent only on their relative displacement(t-s). Based on this property, the dot product calculated for a pair of tokens at positions 5 and 8 would be identical to the dot product for the same pair of tokens if they were located at positions 15 and 18.Diagnosing a Positional Encoding Flaw