In a self-attention mechanism designed for autoregressive tasks, a sequence of 5 tokens is processed. The mechanism computes raw attention scores for each token relative to all other tokens. Before a final normalization step, a mask is added to these scores to prevent any token from attending to future tokens. For the 3rd token in the sequence, which vector correctly represents its scores for all 5 tokens after this causal mask has been applied? (Let s_i denote the original raw score for the 3rd token attending to the i-th token).

Rationale for Causal Mask Values

In a self-attention mechanism processing a sequence of 4 tokens, a mask is added to the raw attention scores to prevent any token from attending to subsequent (future) tokens. Which of the following 4x4 matrices correctly represents this mask?