Next-Token Probability Calculation in Autoregressive Decoders

Enumeration of Dot Products in Causal Self-Attention

A language model is designed to generate text one token at a time, predicting the next token based only on the ones that came before it. The image below shows four possible heatmaps (A, B, C, D) representing the attention scores between tokens in a 4-token sequence. The token making the query is on the vertical axis, and the token providing the key is on the horizontal axis. A darker square indicates that a query token is paying more attention to a key token. Which heatmap correctly illustrates

Debugging a Generative Language Model

Example of Causal Attention Dot Products

Choosing the Right Attention Mechanism