Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling Long-Range Context via Binary Partitioning. arXiv:1911.04070 [cs.CL]

BP Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. ImageTransformer. In Proceedings of ICML.4052–4061. http://proceedings.mlr.press/v80/parmar18a.html

Image Transformer

Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial Attention in Multidimensional Transformers. CoRR abs/1912.12180 (2019). arXiv:1912.12180

Axial Transformer

Typically used for specific data types (block local and 2D are both used for vision data applications)

Carleton College

In positional-based sparse attention, the index set $$G$$ is defined using pre-determined, heuristically designed patterns based on the relative positions of tokens, rather than their content. This means the sparsity pattern is fixed and does not depend on the input values. A common and widely-used example of such a heuristic pattern is the sliding window, where the set $$G$$ for a token at position $$i$$ covers a fixed-size window of nearby tokens.

Positional-based Sparse Attention

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Atomic Sparse Attention Example Diagram

Composed of more than one atomic sparse attention pattern

Compound Sparse Attention

Extended Sparse Attention

An engineer designs a sparse attention mechanism where, for any given token at position `i`, the model is only allowed to attend to the tokens within a fixed-size window around it (e.g., from position `i-k` to `i+k`). This rule is applied uniformly across the entire sequence, irrespective of the specific words involved. Which statement best analyzes the core principle of this design?

A research team is developing a language model for processing extremely long documents. To manage computational costs, they implement an attention strategy where, for any given token, it only attends to (1) the first 50 tokens of the document and (2) the 25 tokens immediately preceding and succeeding it. This pattern is applied consistently to all documents, regardless of their content. Analyze the fundamental principle that defines this attention mechanism and explain why this approach is more computationally efficient than a standard attention mechanism.

Analysis of a Sparse Attention Strategy

In a positional-based sparse attention mechanism, the set of tokens that a given token attends to is dynamically adjusted during processing based on the semantic similarity of the surrounding tokens.

Learn Before

Related

Learn After