Reduces the complexity of attention by reducing the number of queries or key-value pairs

Carleton College

Improvements to the attention module can be categorized into these directions:

1. Sparse Attention
2. Linearized Attention
3. Prototype and Memory Compression
4. Low-rank Self-Attention
5. Attention with Prior
6. Improved Multi-Head Attention

Attention-level improvements of Transformers

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Sparse attention is an efficient alternative to standard self-attention, designed to address its computational and memory challenges. This approach is founded on the principle that for any given token, only a small subset of other tokens in the sequence are contextually important. This implies that most attention weights in a standard attention matrix are close to zero and can be ignored. Consequently, sparse attention models restrict each query to attend to only a limited number of key-value pairs, significantly reducing the computational load.

Sparse Attention

Query Prototyping and Memory Compression

The self-attention **A** $\in \mathbb{R}^{TxT}$  has been observed to be low ranking, meaning that the rank of **A** is far lower than input length 𝑇. This implies that the low-rank property could be explicitly modeled with parameterization. Low rank self-attention is when the self-attention matrix is replaced by a low-rank approximation.

Low Rank Self-Attention

$Prior$ is an attention distribution that comes from a different source than previous from inputs (e.g., softmax(QK$^T$)in vanilla Transformer). Attention with prior is the fusion of two attention distributions, which can be done by computing a weighted sum of the scores corresponding to the prior and generated attention before applying softmax

Attention with Prior

Improved multi-head attention works to introduce more sophisticated mechanisms that guide the behavior of different attention heads or allow interaction across attention heads, as it is not guaranteed that different attention heads indeed capture distinct features in vanilla transformers.

Improved Multi-Head Attention Mechanism

Linear attention is an efficient alternative designed to overcome the memory-intensive limitations of explicitly retaining the entire Key-Value (KV) cache ($$\mathbf{K}_{\le i}$$ and $$\mathbf{V}_{\le i}$$) during the inference of very long sequences. It modifies standard attention by employing a kernel function $$\phi(\cdot)$$ to project each query vector ($$\mathbf{q}_i$$) and key vector ($$\mathbf{k}_i$$) into new representations: $$\mathbf{q}'_i = \phi(\mathbf{q}_i)$$ and $$\mathbf{k}'_i = \phi(\mathbf{k}_i)$$. By applying this transformation and removing the standard Softmax function, the order of matrix multiplications can be rearranged. This structural change avoids the need to compute the large attention matrix and eliminates the requirement to explicitly store the KV cache, making the process highly memory-efficient.

Linear Attention

A research team is working to reduce the computational cost of the attention mechanism for processing extremely long documents. Their proposed solution involves modifying the attention calculation so that each query token only computes attention scores with a small, fixed subset of key tokens (e.g., neighboring tokens and a few globally important tokens) instead of all tokens in the sequence. Which category of attention improvement best describes this approach?

Match each attention improvement strategy with its core operational principle.

A research team is developing a Transformer model for summarizing lengthy scientific articles. They encounter significant memory and computational speed limitations due to the quadratic complexity of the standard attention mechanism. During an analysis of the model's internal states, they consistently find that the attention matrix for any given layer can be closely approximated by a much simpler, lower-dimensional matrix without a significant drop in performance. Which category of attention improvement directly leverages this specific empirical finding, and how does it address the team's performance bottlenecks?

Optimizing Transformer Attention for Long Sequences

A machine learning engineer is tasked with fine-tuning a language model for two different applications: (1) summarizing legal documents, where key information is often found in specific, predictable sections, and (2) analyzing real-time social media feeds, where important context can appear anywhere in a long stream of posts.

The engineer is considering two methods to make the model's attention mechanism more efficient: one that approximates the full attention matrix by assuming it has a low-rank structure, and another that restricts each token to only attend to a predefined, limited set of other tokens.

Analyze the fundamental assumptions behind these two efficiency-improving approaches. Based on your analysis, evaluate which approach is likely to be more effective for each of the two applications and justify your reasoning.

Learn Before

Related