Interpretation of Positional Bias as a Distance Penalty

T5 Bias for Relative Positional Embedding

Shared Learnable Bias per Offset

Heuristic-Based Relative Positional Biases

Comparison of Learned vs. Heuristic-Based Relative Positional Biases

Kerple

FIRE

Relative Position Offset Calculation

A self-attention model incorporates positional awareness by adding a bias term directly to the query-key dot product for each pair of positions (i, j). This bias term's value depends on the relative distance between i and j. What is the primary implication of this approach compared to the alternative of adding positional vectors to the input token embeddings?

Incorporating Positional Bias into Attention Scores

In a self-attention mechanism, the score computed between a query at position i and a key at position j is modified by directly adding a bias term whose value depends only on the positions i and j. What is the primary function of this bias term within the attention calculation?