Short Answer

Analysis of a Positional Encoding Method

Consider an information-processing mechanism where each input item is represented by a 'query' vector and a 'key' vector. Before these vectors are compared, each is rotated in a 2D plane. The angle of rotation for an item at position 'p' is directly proportional to 'p'. The comparison score between a query from position 't' and a key from position 's' is then calculated using the dot product of their respective rotated vectors. A key mathematical property of this operation is that the resulting dot product value is a function of the original, un-rotated vectors and the difference in their positions, (t-s).

Based on this description, explain why this mechanism is said to capture relative positional information rather than absolute positional information.

0

1

Updated 2025-10-03

Contributors are:

Who are from:

Tags

Ch.2 Generative Models - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Analysis in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science