Interpretation of Positional Bias as a Distance Penalty

T5 Bias for Relative Positional Embedding

Shared Learnable Bias per Offset

Heuristic-Based Relative Positional Biases

Comparison of Learned vs. Heuristic-Based Relative Positional Biases

Kerple

FIRE

Relative Position Offset Calculation

A self-attention model incorporates positional awareness by adding a bias term directly to the query-key dot product for each pair of positions (i, j). This bias term's value depends on the relative distance between i and j. What is the primary implication of this approach compared to the alternative of adding positional vectors to the input token embeddings?

Incorporating Positional Bias into Attention Scores

In a self-attention mechanism, the score computed between a query at position i and a key at position j is modified by directly adding a bias term whose value depends only on the positions i and j. What is the primary function of this bias term within the attention calculation?

Shared Learnable Bias per Offset

In a self-attention mechanism that uses relative positioning, consider a sequence of tokens where the model is calculating the attention score. If the current query token is at index 8 and the key token being attended to is at index 5, what is the calculated offset that represents their relative position?

A self-attention model calculates the relative position offset between a query at index i and a key at index j using the formula: offset = i - j. Based on this formula, which of the following conclusions is correct?

In a sequence of tokens, the relative position offset between a query at index i and a key at index j is calculated as i - j. If the query's position i is held constant while the key's position j increases (i.e., the key token appears later in the sequence), how does the calculated offset change?