Concept icon
Concept

Dimension Matching and Projection Shortcuts

The element-wise addition in a residual block requires that the dimensions of the input x\mathbf{x} and the residual output F\mathcal{F} match. When changing channel dimensions or downsampling feature map sizes across stages (performed with a stride of 2), a dimension-matching operation must be applied to the shortcut.

To match dimensions, a linear projection WsW_s is applied along the shortcut connection: y=F(x,{Wi})+Wsx\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + W_s\mathbf{x} Three shortcut strategies are evaluated for handling dimension changes:

  • Option A: Zero-padding shortcuts are used to increase dimensions, keeping all shortcuts parameter-free.
  • Option B: Projection shortcuts (implemented via 1×11 \times 1 convolutions) are used only when increasing dimensions, while all other shortcuts remain parameter-free identity mappings.
  • Option C: All shortcuts in the network are linear projections.

While all three options substantially outperform plain baselines, Option B achieves slightly better accuracy than Option A because zero-padded entries do not participate in residual learning. Option C yields only a marginal improvement over Option B at the expense of adding many extra parameters. Because projection shortcuts are not essential for solving the degradation problem, identity shortcuts are preferred to minimize model size and computational complexity.

0

1

Concept icon
Updated 2026-09-07

Tags

Prep Sessions

Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Ch.3 Deep Residual Network Architecture - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Residual Formulation and Shortcut Connections - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Identity versus Projection Shortcuts - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

Ch.4 Residual Network Experiments and Applications - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor

ImageNet Classification and Model Variations - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor