Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial Attention in Multidimensional Transformers. CoRR abs/1912.12180 (2019). https://arxiv.org/abs/1912.12180

Carleton College

Google

Typically used for specific data types (block local and 2D are both used for vision data applications)

Extended Sparse Attention

Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling Long-Range Context via Binary Partitioning. arXiv:1911.04070 [cs.CL]. https://arxiv.org/abs/1911.04070

BP Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. ImageTransformer. In Proceedings of ICML.4052–4061. http://proceedings.mlr.press/v80/parmar18a.html

Learn Before

Related