In a Transformer decoder, masked self-attention is used to ensure that the prediction for a token at a given position can only depend on previous tokens. This is achieved by modifying the attention score matrix before the softmax function is applied. For a sequence of tokens, which of the following correctly describes the structure of the attention score matrix after this causal mask has been applied?

A Transformer decoder is calculating its output for a specific token in a sequence. To ensure it only uses information from that token and previous tokens, it employs a special attention mechanism. Arrange the following five operations in the correct chronological order as they would occur within this mechanism.

In masked self-attention, which key positions is the query of a given token permitted to interact with?

Autoregressive Dependency Guarantee in Transformer Decoders

At which stage in the masked self-attention mechanism is the mask applied to zero out weights for future tokens?