Formula for Attention Score with ALiBi Bias

Linear Relative Position Bias Example

In a sequence processing model, a positional bias is calculated to penalize attention scores based on the distance between tokens. The formula used is Bias = -β ⋅ (i - j), where i is the query position, j is the key position, and β is a fixed scalar. If the query token is at position 5, the key token is at position 2, and β = 0.1, what is the calculated bias value?

Visual Example of a Linear Relative Position Bias in Causal Attention

True or False: According to the positional bias formula PE(i, j) = -β ⋅ (i - j), where i is the query position, j is the key position, and β is a positive scalar, the penalty applied to the attention score decreases as the distance between the query and key tokens increases.

Interpreting a Linear Positional Bias Value

Similarity of ALiBi Positional Biases to Length Features