Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. ImageTransformer. In Proceedings of ICML.4052–4061. http://proceedings.mlr.press/v80/parmar18a.html

Carleton College

Typically used for specific data types (block local and 2D are both used for vision data applications)

Extended Sparse Attention

Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling Long-Range Context via Binary Partitioning. arXiv:1911.04070 [cs.CL]

BP Transformer

Image Transformer

Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial Attention in Multidimensional Transformers. CoRR abs/1912.12180 (2019). arXiv:1912.12180

Learn Before

Related