These are transformers designed to alleviate overhead, and make the transformer 'lightweight' in terms of model size and computation

A few examples are:
- Lite Transformer
- Funnel Transformer
- DeLighT

Lightweight transformers

The main idea of this concept is to create a forward path between adjacent Transformer blocks. A few methods of implementing this are:

- reuse attention distributions from previous block to guide attention of current block
- use a weighted sum of encoder representations at all encoder layers
- add a feedback mechanism to Transformer decoder, where each position attends to a weighted sum of history representations from all layer

Strengthening Cross-Block Connectivity in transformers

An intriguing and promising modification is to make computation time conditioned on the inputs, i.e., to introduce Adaptive Computation Time (ACT), as opposed to the fixed computation procedure used in vanilla transformers. This allows for a deeper and more refined representation for complex inputs, and a shallow, more efficient representation for easier inputs

Adaptive Computation Time (ACT) in transformers

An input sequence can be decomposed into finer segments that can be efficiently processed by Transformer or Transformer modules

Divide-and-Conquer Strategies in transformers

These are high-level adaptations to transformers, past the module-level

- Lightweight Transformers
- Strengthening Cross-Block Connectivity
- Adaptive Computation Time (ACT)
- Divide-and-Conquer Strategies

Carleton College

X-formers can be categorized into:

1. Module-level
2. Architecture-level
3. Pre-training methods
4. Applications

Categorization of Transformer variants

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Taxonomy of Transformer variants

Variant with an augmentation to a specific module of the transformer model

Module-Level variant

Architecture-level variants of transformers

1. NLP
	- machine translation
	- language modeling
	- named entity recognition

2. Computer Vision
	- image classification
	- object detection
	- image generation
	- video processing

3. Audio Applications
	- speech recognition
	- speech synthesis
	- speech enhancement
	- music generation

4. Multimodal applications
	- Visual question-answering
	- visual common-sense reasoning
	- caption generation
	- speech-to-text translation
	- text-to-image generation

Applications of transformers

The pre-training and fine-tuning paradigm is a method motivated by the goal of creating adaptable, general-purpose systems for universal language understanding and generation. It involves separating the common components of neural network architectures, such as Transformers, and training them on vast amounts of unlabeled data using self-supervision. The resulting systems, known as foundation models, can be easily adapted for specific downstream applications via fine-tuning or prompting. This paradigm shift has enormously transformed natural language processing, meaning that in many cases, large-scale supervised learning for specific tasks is no longer required.

Learn Before

Related

Learn After