University of Michigan - Ann Arbor

Although the linear transformations in the feed-forward network are executed identically across all token positions within a single layer, the parameters ($W_1, b_1, W_2, b_2$) differ from layer to layer throughout the model stack.

This position-wise operation can alternatively be described as two consecutive convolutions with a kernel size of 1. Furthermore, combining a self-attention layer with a point-wise feed-forward layer exhibits the same computational complexity as a separable convolution where the kernel size $k$ equals the sequence length $n$.

Within both the encoder and decoder, the position-wise feed-forward sub-layer is integrated using residual connections followed by layer normalization, yielding an output of $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$.

Properties and Convolutional Interpretation of the FFN

Match each architectural property or interpretation of the position-wise feed-forward network to its corresponding description.

Evaluate the engineer's implementation against the standard design of the position-wise feed-forward network. Identify the two errors in how parameters are distributed across token positions and layers, and explain how the weights must actually be configured.

Learn Before

Related