Positionwise Nature of Transformer Feed-Forward Networks
In a Transformer architecture, the feed-forward network is called positionwise because it applies the identical Multi-Layer Perceptron (MLP) to transform the representation at every sequence position independently. For an input tensor with the shape (batch size, number of time steps, number of hidden units), this two-layer MLP processes each time step's vector in isolation. Consequently, only the innermost dimension is transformed, resulting in an output tensor of shape (batch size, number of time steps, ). Because the exact same MLP transforms all positions, identical inputs at different positions will produce identical outputs.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Transformer Encoder-Decoder Architecture - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Feed-Forward Network (FFN) Formula and Component Dimensions in Transformers
An engineer is building a deep neural network for sequence processing. Each layer of the network consists of a self-attention mechanism followed by a position-wise sub-layer. The engineer designs this position-wise sub-layer to be composed of two consecutive linear transformations. What is the most significant negative consequence of omitting a non-linear activation function between these two linear transformations?
A researcher modifies the position-wise sub-layer within a sequence processing model. The standard design for this sub-layer is a sequence of: a linear transformation, a non-linear activation, and a second linear transformation. The researcher's modification adds a second non-linear activation function immediately after the final linear transformation. Which of the following best evaluates the impact of this architectural change?
FFN Hidden Size in Transformers
Positionwise Nature of Transformer Feed-Forward Networks
MLP of the Vision Transformer Encoder
In a standard Transformer Feed-Forward Network (FFN), the first fully connected layer typically utilizes a non-linear activation function such as ReLU.
Explain the primary purpose of the Feed-Forward Network (FFN) sub-layer in Transformer models, detailing how its function affects representation learning and supports the self-attention mechanism.
In the standard architecture of a Transformer Feed-Forward Network (FFN), what specific type of layer is employed as the second fully connected layer?
Transformer Encoder–Decoder Architecture
Transformer Encoder Sublayers
Transformer Decoder
Positionwise Nature of Transformer Feed-Forward Networks
Learn After
Match each architectural concept of the Transformer feed-forward layer with its correct functional description.
Determine the shape of the output tensor produced by this feed-forward network, identify which specific dimension of tensor X is transformed, and explain why the remaining dimensions remain unchanged.