Learn Before
Under masked self-attention, which key positions is a query for a given token permitted to interact with?
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Scaled Dot-Product Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
In a Transformer decoder, masked self-attention is used to ensure that the prediction for a token at a given position can only depend on previous tokens. This is achieved by modifying the attention score matrix before the softmax function is applied. For a sequence of tokens, which of the following correctly describes the structure of the attention score matrix after this causal mask has been applied?
In masked self-attention, which key positions is the query of a given token permitted to interact with?
Autoregressive Dependency Guarantee in Transformer Decoders
In standard self-attention, each position is restricted from attending to subsequent tokens in the sequence.
Under masked self-attention, which key positions is a query for a given token permitted to interact with?
What core capability does masked self-attention enable in the Transformer decoder?
Order the operations performed during masked self-attention from first to last: