Calculating Attention Weights (αi,j) in Transformers

Relative Positional Encoding as a Query-Key Bias

In a sequence processing model, an intermediate score is calculated to determine the relationship between two elements. This score is found by taking the dot product of a 'query' vector and a 'key' vector, and then scaling the result by dividing by the square root of the vectors' dimension. Assume no other adjustments are made to the score.

Given the following information:

• Query vector: [2.0, 0.5, 1.0, -1.5]
• Key vector: [1.0, 1.0, -0.5, 2.0]
• Vector dimension: 4

What is the calculated intermediate score?

In a transformer model designed for text generation, a masking mechanism is applied to the attention scores (βi,j) to prevent a token at position i from attending to future tokens (positions j > i). This is achieved by adding a large negative number (e.g., -∞) to the score before normalization. Consider the calculation of attention scores for a sequence of 4 tokens. Which of the following matrices correctly represents the application of this causal mask, where 'Score' indicates a calculated value and '-∞' indicates a masked value?

Analyzing Training Instability in an Attention Mechanism