Value Matrix for Causal Attention (V_≤i)

Key Matrix from a Sliding Window

Consider the following three row vectors: r_1 = [5, 0, 3], r_2 = [1, 2, 8], and r_3 = [4, 7, 6]. If a matrix M is constructed by vertically stacking these vectors in the order r_1, r_2, then r_3 (with r_1 as the top row), what is the resulting matrix M?

A matrix M is formed by vertically stacking its row vectors, m_0, m_1, and m_2. Given the matrix M shown below, identify the row vector m_1.

$M = \begin{bmatrix} 8 & 1 & 6 \\ 3 & 5 & 7 \\ 4 & 9 & 2 \end{bmatrix}$

A matrix A is constructed by vertically stacking four row vectors, where each row vector contains five elements. The resulting matrix A will have 5 rows and 4 columns.

Key Matrix from a Sliding Window

Value Matrix from a Sliding Window

An engineer is optimizing a language model that processes long documents using an attention mechanism that considers a fixed-size window of the most recent tokens. If the engineer decides to significantly increase the size of this window, what is the primary trade-off they will encounter?

Determining the Context Window

Diagnosing Long-Range Dependency Failures

Key Matrix from a Sliding Window

Consider a matrix $\mathbf{M}$ that contains 10 row vectors, indexed from 1 to 10 (i.e., $\mathbf{m}_1, \mathbf{m}_2, ..., \mathbf{m}_{10}$). The notation $\mathbf{M}_{[a, b]}$ is used to select a sub-matrix by vertically stacking the row vectors from index $a$ to index $b$, inclusive. Which of the following sub-matrices correctly represents $\mathbf{M}_{[6, 9]}$?

The notation $\mathbf{M}_{[j-9, j]}$ is used to select a slice of row vectors from a larger matrix $\mathbf{M}$. This slice contains a total of ____ row vectors.

Extracting a Context Window from a Token Matrix