The basic concept behind this is to replace FFNs with similar structures with much more parameters, which supports a larger model capacity

Carleton College

Here are a few modifications proposed to improve the feed-forward network (FFN) of the transformer:


- changes to the activation function
- Adapting the FFN for Larger Capacity
- Dropping FFN Layers

Improvements to the FFN of a transformer

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Vanilla transformers use ReLU activation. Other functions used as activation functions include:

- Swish function -> f(x) = *x*sigmoid($\beta$ x)
- Gaussian Error Linear Unit (GELU)
- Gated Linear Units (GLU)

Activation function of the FFN in transformers

Adapting the FFN for Larger Capacity in transformers

It has been shown that the FFN layer can be mitigated completely without loss of performance, which decreases the complexity of the model

Learn Before

Related