Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite Transformer with Long-Short Range Attention. In Proceedings of ICLR. https://openreview.net/forum?id=ByeMPlHKPH

Carleton College

These are transformers designed to alleviate overhead, and make the transformer 'lightweight' in terms of model size and computation

A few examples are:
- Lite Transformer
- Funnel Transformer
- DeLighT

Lightweight transformers

Lite Transformer

Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing. In Proceedings of NeurIPS. https://proceedings.neurips.cc/paper/2020/hash/
2cd2915e69546904e4e5d4a2ac9e1652-Abstract.html

Funnel Transformer

Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2020. DeLighT: Very Deep and Light-weight Transformer. arXiv:2008.00623 [cs.LG]

Learn Before

Related