Learn Before
Low-Rank Self-Attention
The self-attention matrix has been observed to have a low rank, meaning that the rank of is far lower than the input sequence length . This implies that the low-rank property can be explicitly modeled with parameterization. Low-rank self-attention is an efficiency improvement where the standard self-attention matrix is replaced by a low-rank approximation.
0
1
Tags
Data Science
Related
Sparse Attention
Query Prototyping and Memory Compression
Attention with Prior
Improved Multi-Head Attention Mechanism
Linear Attention
A research team is working to reduce the computational cost of the attention mechanism for processing extremely long documents. Their proposed solution involves modifying the attention calculation so that each query token only computes attention scores with a small, fixed subset of key tokens (e.g., neighboring tokens and a few globally important tokens) instead of all tokens in the sequence. Which category of attention improvement best describes this approach?
Match each attention improvement strategy with its core operational principle.
Optimizing Transformer Attention for Long Sequences
Evaluating Attention Optimization Strategies for Specific Applications
Low-Rank Self-Attention