Properties and Convolutional Interpretation of the FFN
Although the linear transformations in the feed-forward network are executed identically across all token positions within a single layer, the parameters () differ from layer to layer throughout the model stack.
This position-wise operation can alternatively be described as two consecutive convolutions with a kernel size of 1. Furthermore, combining a self-attention layer with a point-wise feed-forward layer exhibits the same computational complexity as a separable convolution where the kernel size equals the sequence length .
Within both the encoder and decoder, the position-wise feed-forward sub-layer is integrated using residual connections followed by layer normalization, yielding an output of .
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Position-Wise Feed-Forward Networks - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
Match each architectural property or interpretation of the position-wise feed-forward network to its corresponding description.
Evaluate the engineer's implementation against the standard design of the position-wise feed-forward network. Identify the two errors in how parameters are distributed across token positions and layers, and explain how the weights must actually be configured.