Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling Long-Range Context via Binary Partitioning. arXiv:1911.04070 [cs.CL]. https://arxiv.org/abs/1911.04070

Carleton College

Google

Typically used for specific data types (block local and 2D are both used for vision data applications)

Extended Sparse Attention

BP Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. ImageTransformer. In Proceedings of ICML.4052–4061. http://proceedings.mlr.press/v80/parmar18a.html

Image Transformer

Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial Attention in Multidimensional Transformers. CoRR abs/1912.12180 (2019). https://arxiv.org/abs/1912.12180

Learn Before

Related