Scaled Dot-Product Attention

Causal Self-Attention in Autoregressive Decoders

A model is processing a sequence of three tokens. For the query at position 2, the un-normalized attention scores with respect to the keys at positions 0, 1, and 2 are calculated as [1.0, 2.0, 3.0] respectively. What is the final attention weight that the token at position 2 will assign to the token at position 1?

Attention Output as a Weighted Sum of Values

Impact of Masking on Attention Weight Distribution

True or False: In a self-attention mechanism, if you add the same constant value to all un-normalized attention scores corresponding to a single query vector, the final normalized attention weights for that query will change.

Attention Weight Formula ($\alpha_{i,j}$)

Single-Query Attention Computation with Multiplicative Scaling

Scaled Dot-Product Attention

General Attention Formula

Value Matrix for Causal Attention (V_≤i)

Value Matrix from a Sliding Window

An attention mechanism processes an input sequence of 20 tokens, where each token is represented by a 256-dimensional vector. A Value matrix (V) is generated as part of this process. Which of the following statements most accurately describes the properties and role of this V matrix?

Determining Value Matrix Dimensions

Debugging an Attention Mechanism

Scaled Dot-Product Attention

Multi-Head Self-Attention Function

Purpose and Structure of the Feed-Forward Network (FFN) in Transformers

A standard processing block in a Transformer model consists of two main sub-layers applied in sequence. The first sub-layer's primary role is to relate different positions of the input sequence to compute a new representation for each position. The second sub-layer then applies an identical non-linear transformation to each position's representation independently. How does the core computational function, denoted as F(·), implemented within each of these sub-layers, differ?

A standard processing block in a certain neural network architecture consists of two main sub-layers. Each sub-layer's computation can be described as applying a core function, F(·), within a structure that also includes a residual connection and layer normalization. Match each sub-layer type with the correct description of its core computational function, F(·).

Identifying Core Functions in a Transformer Block