Mechanism of Relative Positional Embedding
The fundamental mechanism behind relative positional embedding is to determine the offset—or the relative distance—between any given pair of token positions. Once this positional offset is obtained, it is directly incorporated into the self-attention model, allowing the system to utilize pairwise token relationships rather than absolute positions.
0
1
Tags
Foundations of Large Language Models
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A language model is trained exclusively on text sequences with a maximum length of 512 tokens. During evaluation, the model shows a significant drop in performance when processing documents that are 1000 tokens long. The engineers hypothesize the problem is related to how the model incorporates word order information. Which of the following changes to the model's architecture is most likely to resolve this specific issue?
Positional Encoding for Machine Translation
Positional Invariance in Self-Attention
Mechanism of Relative Positional Embedding
Mechanism of Relative Positional Embedding