Improvements to the attention module can be categorized into these directions:

1. Sparse Attention
2. Linearized Attention
3. Prototype and Memory Compression
4. Low-rank Self-Attention
5. Attention with Prior
6. Improved Multi-Head Attention

Attention-level improvements of Transformers

These are mechanisms that inject positional information into Transformers. Positional encodings are important within the transformer architecture because the ordering of words matters when modeling sequences of text.


Types of Positional Representations:
- absolute positional representations
- relative position representations
- implicit representations
- other representations

Positional Representations of Transformers

Here are a few modifications proposed to improve the feed-forward network (FFN) of the transformer:


- changes to the activation function
- Adapting the FFN for Larger Capacity
- Dropping FFN Layers

Improvements to the FFN of a transformer

Layer normalization (LN) is a widely-used architectural modification that is critical for stabilizing the training of deep networks like Transformers. It operates by normalizing the inputs across all features for each training example independently. The specific mathematical function used for layer normalization is central to its application. Key areas of research and improvement for LN in transformers include its placement within the architecture, the development of effective substitutes, and the creation of normalization-free models.

Layer Normalization in Transformers

Variant with an augmentation to a specific module of the transformer model

Carleton College

X-formers can be categorized into:

1. Module-level
2. Architecture-level
3. Pre-training methods
4. Applications

Categorization of Transformer variants

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Taxonomy of Transformer variants

Module-Level variant

These are high-level adaptations to transformers, past the module-level

- Lightweight Transformers
- Strengthening Cross-Block Connectivity
- Adaptive Computation Time (ACT)
- Divide-and-Conquer Strategies

Architecture-level variants of transformers

1. NLP
	- machine translation
	- language modeling
	- named entity recognition

2. Computer Vision
	- image classification
	- object detection
	- image generation
	- video processing

3. Audio Applications
	- speech recognition
	- speech synthesis
	- speech enhancement
	- music generation

4. Multimodal applications
	- Visual question-answering
	- visual common-sense reasoning
	- caption generation
	- speech-to-text translation
	- text-to-image generation

Applications of transformers

The pre-training and fine-tuning paradigm is a method motivated by the goal of creating adaptable, general-purpose systems for universal language understanding and generation. It involves separating the common components of neural network architectures, such as Transformers, and training them on vast amounts of unlabeled data using self-supervision. The resulting systems, known as foundation models, can be easily adapted for specific downstream applications via fine-tuning or prompting. This paradigm shift has enormously transformed natural language processing, meaning that in many cases, large-scale supervised learning for specific tasks is no longer required.

Learn Before

Related

Learn After