Role of Causal Attention in Autoregressive Language Models

Causal Attention Output for a Single Token

Visualization of Query-Key Dot Products in Causal Attention

An autoregressive model calculates a square attention weight matrix using the formula: Softmax((QK^T / sqrt(d)) + Mask). The purpose of the Mask component is to prevent any token from attending to subsequent tokens in the sequence. Which statement best describes the resulting attention weight matrix?

An autoregressive model is processing a sequence of 4 tokens. To ensure that the prediction for any given token is based only on the tokens that came before it and the token itself, a specific structure is imposed on the attention weight matrix. Which of the following 4x4 matrices correctly illustrates this structure, where 'α' represents a calculated, non-zero attention weight and '0' represents a weight that has been forcibly set to zero?

Applying a Causal Mask to Attention Scores