It has been shown that the FFN layer can be mitigated completely without loss of performance, which decreases the complexity of the model

Carleton College

Here are a few modifications proposed to improve the feed-forward network (FFN) of the transformer:


- changes to the activation function
- Adapting the FFN for Larger Capacity
- Dropping FFN Layers

Improvements to the FFN of a transformer

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Vanilla transformers use ReLU activation. Other functions used as activation functions include:

- Swish function -> f(x) = *x*sigmoid($\beta$ x)
- Gaussian Error Linear Unit (GELU)
- Gated Linear Units (GLU)

Activation function of the FFN in transformers

The basic concept behind this is to replace FFNs with similar structures with much more parameters, which supports a larger model capacity

Learn Before

Related