In Transformer-based systems, the post-norm architecture is a specific sub-layer configuration in which layer normalization is performed right after a residual block.

Google

In vanilla transformers, the LN layer lies between the residual blocks, called *post-LN*. An improvement to this is called *pre-LN*, which is when the LN layer is placed inside the residual connection before the attention or FFN, with an additional LN after the final layer to control the magnitude of final outputs. This has shown to eliminate the need for learning-rate warm-up stage.

Placement of Layer Normalization in transformers

Reference of Foundations of Large Language Models Course

Post-Norm Architecture in Transformers

In Transformer-based systems, the pre-norm architecture is a specific sub-layer configuration where layer normalization is applied internally within a residual block. Because this approach is remarkably effective at stabilizing the training of deep neural networks, it serves as the underlying structural basis for the majority of modern Large Language Models.

Learn Before

Related