Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-Transformer. In Proceedings of HLT-NAACL.1315–1325. https://www.aclweb.org/anthology/N19-1133

Star-Transformer

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150 [cs.CL]

Longformer

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding Long and Structured Inputs in Transformers. In Proceedings of EMNLP. Online, 268–284. https://doi.org/10.18653/v1/2020.emnlp-main.19

Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. arXiv:2007.14062 [cs.LG]

BigBird Transformer

Composed of more than one atomic sparse attention pattern

Carleton College

In positional-based sparse attention, the index set $$G$$ is defined using pre-determined, heuristically designed patterns based on the relative positions of tokens, rather than their content. This means the sparsity pattern is fixed and does not depend on the input values. A common and widely-used example of such a heuristic pattern is the sliding window, where the set $$G$$ for a token at position $$i$$ covers a fixed-size window of nearby tokens.

Positional-based Sparse Attention

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Atomic Sparse Attention Example Diagram

Compound Sparse Attention

Typically used for specific data types (block local and 2D are both used for vision data applications)

Extended Sparse Attention

An engineer designs a sparse attention mechanism where, for any given token at position `i`, the model is only allowed to attend to the tokens within a fixed-size window around it (e.g., from position `i-k` to `i+k`). This rule is applied uniformly across the entire sequence, irrespective of the specific words involved. Which statement best analyzes the core principle of this design?

A research team is developing a language model for processing extremely long documents. To manage computational costs, they implement an attention strategy where, for any given token, it only attends to (1) the first 50 tokens of the document and (2) the 25 tokens immediately preceding and succeeding it. This pattern is applied consistently to all documents, regardless of their content. Analyze the fundamental principle that defines this attention mechanism and explain why this approach is more computationally efficient than a standard attention mechanism.

Analysis of a Sparse Attention Strategy

In a positional-based sparse attention mechanism, the set of tokens that a given token attends to is dynamically adjusted during processing based on the semantic similarity of the surrounding tokens.

Learn Before

Related

Learn After