In a Transformer decoder, masked self-attention is used to ensure that the prediction for a token at a given position can only depend on previous tokens. This is achieved by modifying the attention score matrix before the softmax function is applied. For a sequence of tokens, which of the following correctly describes the structure of the attention score matrix after this causal mask has been applied?

In masked self-attention, which key positions is the query of a given token permitted to interact with?

Autoregressive Dependency Guarantee in Transformer Decoders

In standard self-attention, each position is restricted from attending to subsequent tokens in the sequence.

Under masked self-attention, which key positions is a query for a given token permitted to interact with?

What core capability does masked self-attention enable in the Transformer decoder?

Order the operations performed during masked self-attention from first to last: