Sparse Attention

Query Prototyping and Memory Compression

Low Rank Self-Attention

Attention with Prior

Improved Multi-Head Attention Mechanism

Linear Attention

A research team is working to reduce the computational cost of the attention mechanism for processing extremely long documents. Their proposed solution involves modifying the attention calculation so that each query token only computes attention scores with a small, fixed subset of key tokens (e.g., neighboring tokens and a few globally important tokens) instead of all tokens in the sequence. Which category of attention improvement best describes this approach?

Match each attention improvement strategy with its core operational principle.

Optimizing Transformer Attention for Long Sequences

Evaluating Attention Optimization Strategies for Specific Applications

Classification of Long Sequence Modeling Problems

Increased Research Interest in Long-Context LLMs

Long-Context LLMs

Research Directions for Adapting Transformers to Long Contexts

Sparse Attention

Challenges in Training and Deploying High-Capacity Models

Challenge of Streaming Context for LLMs

Key Issues in Long-Context Language Modeling Methods

Challenge of Training New Architectures for Long-Context LLMs

Key Techniques for Long-Input Adaptation in LLMs

RoPE Scaling Transformation Equivalence

Architectural Prioritization for a Long-Context LLM

A development team is attempting to use a standard Transformer-based LLM for real-time analysis of continuous data streams, where the input sequence can grow to hundreds of thousands of tokens. They encounter two main problems: the time it takes to process each new token increases dramatically as the sequence gets longer, and the system frequently runs out of memory. Which statement correctly analyzes the architectural sources of these two distinct problems?

Differentiating Bottlenecks in Long-Sequence LLMs

Attention Weight Matrix (α)

Sparse Attention

Self-attention layers' first approach

In a general attention mechanism, the output is calculated as a weighted sum of the Value vectors, where the weights are determined by the interaction between Query and Key vectors. The standard formula is: $Att(\textbf{Q}, \textbf{K}, \textbf{V}) = \alpha(\textbf{Q}, \textbf{K})\textbf{V}$. Consider a scenario where this formula is mistakenly altered to be: $Att_{modified} = \alpha(\textbf{Q}, \textbf{K})\textbf{K}$. What is the most significant consequence of this modification?

Dimensional Analysis of the Attention Formula

Applying the Attention Mechanism Roles

Self-Attention Output Formula for a Single Query