Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite Transformer with Long-Short Range Attention. In Proceedings of ICLR. https://openreview.net/forum?id=ByeMPlHKPH

Lite Transformer

Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing. In Proceedings of NeurIPS. https://proceedings.neurips.cc/paper/2020/hash/
2cd2915e69546904e4e5d4a2ac9e1652-Abstract.html

Funnel Transformer

Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2020. DeLighT: Very Deep and Light-weight Transformer. arXiv:2008.00623 [cs.LG]

DeLighT

These are transformers designed to alleviate overhead, and make the transformer 'lightweight' in terms of model size and computation

A few examples are:
- Lite Transformer
- Funnel Transformer
- DeLighT

Carleton College

These are high-level adaptations to transformers, past the module-level

- Lightweight Transformers
- Strengthening Cross-Block Connectivity
- Adaptive Computation Time (ACT)
- Divide-and-Conquer Strategies

Architecture-level variants of transformers

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Lightweight transformers

The main idea of this concept is to create a forward path between adjacent Transformer blocks. A few methods of implementing this are:

- reuse attention distributions from previous block to guide attention of current block
- use a weighted sum of encoder representations at all encoder layers
- add a feedback mechanism to Transformer decoder, where each position attends to a weighted sum of history representations from all layer

Strengthening Cross-Block Connectivity in transformers

An intriguing and promising modification is to make computation time conditioned on the inputs, i.e., to introduce Adaptive Computation Time (ACT), as opposed to the fixed computation procedure used in vanilla transformers. This allows for a deeper and more refined representation for complex inputs, and a shallow, more efficient representation for easier inputs

Adaptive Computation Time (ACT) in transformers

An input sequence can be decomposed into finer segments that can be efficiently processed by Transformer or Transformer modules

Learn Before

Related

Learn After