Google

In the context of sliding window attention and sequence processing, $n_c$ is a parameter that denotes the size of the window. It specifies the number of recent elements or tokens to be included in the current context.

Window Size (n_c)

A key matrix from a sliding window is a sub-matrix formed by selecting a contiguous sequence of key vectors. When denoted using slice notation as $\mathbf{K}_{[i-n_c+1,i]}$, it is constructed by vertically stacking the key vectors from index $i-n_c+1$ up to $i$. This structure is represented by the formula: $$\mathbf{K}_{[i-n_c+1,i]} = \begin{bmatrix} \mathbf{k}_{i-n_c+1} \\ \vdots \\ \mathbf{k}_i \end{bmatrix}$$ This matrix is a fundamental component in attention mechanisms that operate on a fixed-size context window.

Key Matrix from a Sliding Window

The notation $\mathbf{V}_{[i-n_c+1,i]}$ represents a matrix created by vertically stacking a sequence of value vectors within a sliding window. This matrix contains the value vectors from index $i-n_c+1$ to the current index $i$, capturing the $n_c$ most recent values. It is formally defined as: $$\mathbf{V}_{[i-n_c+1,i]} = \begin{bmatrix} \mathbf{v}_{i-n_c+1} \\ \vdots \\ \mathbf{v}_i \end{bmatrix}$$

Value Matrix from a Sliding Window

An engineer is optimizing a language model that processes long documents using an attention mechanism that considers a fixed-size window of the most recent tokens. If the engineer decides to significantly increase the size of this window, what is the primary trade-off they will encounter?

A language model processes a sequence of tokens one by one. To compute the representation for the current token, it uses an attention mechanism that only considers a local context. This context consists of the current token plus a fixed number of the most recent preceding tokens. The total number of tokens in this context is defined by a 'window size' parameter.

Given the sequence `[T1, T2, T3, T4, T5, T6, T7, T8, T9, T10]` and a window size of 4, which specific tokens form the context when the model is processing token `T8`?

Determining the Context Window

Read the following scenario and identify the most likely parameter-related cause of the model's failure, explaining your reasoning.

Learn Before

Related