The structure of the Encoder is shown in the figure

Northeastern University (US)

Transformers are sequence-to-sequence models that consist of an encoder and a decoder, each of which is a stack of *L* identical blocks

Each encoder block is composed of a multi-head self-attention module and position wise feed-forward network (FFN).

Each decoder block is also composed of a multi-head self-attention module and FFN, but with the added components of having cross-attention from the encoder and masked self-attention

Overview of a Transformer

A very influential paper that introduced the concept of Transformer model.
https://arxiv.org/abs/1706.03762

Attention Is All You Need

Encoder Structure of Transformer

The structure of the Decoder is shown in the figure

Decoder Structure  of Transformer

In Transformer models, the Feed-Forward Network (FFN) sub-layer plays a crucial role by introducing non-linearities into the representation learning process. This function is vital for preventing the representations learned by the self-attention mechanism from degenerating. Structurally, a standard FFN consists of two fully connected layers. The first layer typically uses a non-linear activation function like ReLU, while the second is a linear layer.

Purpose and Structure of the Feed-Forward Network (FFN) in Transformers

The self-attention mechanism, a core component of Transformer models, is a significant contributor to the challenges and complexity encountered during the inference process.

Learn Before

Related