Vanilla transformers use ReLU activation. Other functions used as activation functions include:

- Swish function -> f(x) = *x*sigmoid($\beta$ x)
- Gaussian Error Linear Unit (GELU)
- Gated Linear Units (GLU)

Activation function of the FFN in transformers

The basic concept behind this is to replace FFNs with similar structures with much more parameters, which supports a larger model capacity

Adapting the FFN for Larger Capacity in transformers

It has been shown that the FFN layer can be mitigated completely without loss of performance, which decreases the complexity of the model

 Dropping FFN Layers in transformers

Here are a few modifications proposed to improve the feed-forward network (FFN) of the transformer:


- changes to the activation function
- Adapting the FFN for Larger Capacity
- Dropping FFN Layers

Carleton College

Variant with an augmentation to a specific module of the transformer model

Module-Level variant

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Improvements to the attention module can be categorized into these directions:

1. Sparse Attention
2. Linearized Attention
3. Prototype and Memory Compression
4. Low-rank Self-Attention
5. Attention with Prior
6. Improved Multi-Head Attention

Attention-level improvements of Transformers

These are mechanisms that inject positional information into Transformers. Positional encodings are important within the transformer architecture because the ordering of words matters when modeling sequences of text.


Types of Positional Representations:
- absolute positional representations
- relative position representations
- implicit representations
- other representations

Positional Representations of Transformers

Improvements to the FFN of a transformer

Layer normalization (LN) is a widely-used architectural modification that is critical for stabilizing the training of deep networks like Transformers. It operates by normalizing the inputs across all features for each training example independently. The specific mathematical function used for layer normalization is central to its application. Key areas of research and improvement for LN in transformers include its placement within the architecture, the development of effective substitutes, and the creation of normalization-free models.

Learn Before

Related

Learn After