Complete ALiBi Attention Formula

Calculating a Pre-Softmax Attention Score with Linear Bias

In a model that adds a linear positional bias to its attention calculation, a query at position i=10 attends to two keys at positions j1=5 and j2=2. Assuming the scaled dot-product portion of the score is identical for both keys, how will the addition of the positional bias term PE(i, j) affect their final pre-Softmax attention scores?

Interaction of Semantic and Positional Scores