Autoregressive Dependency Guarantee in Transformer Decoders
To preserve autoregression in the Transformer decoder, the prediction generated for position must depend exclusively on known outputs at positions strictly less than . This constraint is enforced by two synchronized mechanisms: modifying the decoder's self-attention sub-layer to mask out subsequent positions, and shifting the output embeddings to the right by one position.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture Fundamentals - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Decoder Masking and Autoregressive Prediction - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Related
Autoregressive Dependency Guarantee in Transformer Decoders
Masked Self-Attention in Transformer Decoders
In a Transformer decoder, masked self-attention is used to ensure that the prediction for a token at a given position can only depend on previous tokens. This is achieved by modifying the attention score matrix before the softmax function is applied. For a sequence of tokens, which of the following correctly describes the structure of the attention score matrix after this causal mask has been applied?
A Transformer decoder is calculating its output for a specific token in a sequence. To ensure it only uses information from that token and previous tokens, it employs a special attention mechanism. Arrange the following five operations in the correct chronological order as they would occur within this mechanism.
In masked self-attention, which key positions is the query of a given token permitted to interact with?
Autoregressive Dependency Guarantee in Transformer Decoders
At which stage in the masked self-attention mechanism is the mask applied to zero out weights for future tokens?
Learn After
To preserve autoregression in the Transformer decoder, what dependency constraint must be satisfied by the prediction generated for position ?
What positional adjustment must be applied to the output embeddings to help enforce the autoregressive constraint in the Transformer decoder?
Explain how the Transformer decoder prevents future position information from influencing predictions, detailing the necessary mechanisms and the operational condition they uphold.