Learn Before
Residual Formulation and Shortcut Connections - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Identity versus Projection Shortcuts - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Deeper Bottleneck Architectures - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Bottleneck Residual Blocks
To build extremely deep networks such as ResNet-50, ResNet-101, and ResNet-152 within feasible training times, the standard two-layer residual block is modified into a three-layer bottleneck block.
In a bottleneck block, the residual function consists of a stack of three convolutions: , , and . The first convolution is responsible for reducing the channel dimensions (e.g., from 256 to 64), the middle convolution operates on this lower-dimensional bottleneck, and the final convolution restores the dimensions back to the high-dimensional representation (e.g., restoring back to 256).
Parameter-free identity shortcuts are essential for bottleneck architectures. Because the shortcut connects the two high-dimensional ends of the block, replacing the identity shortcut with a projection shortcut would double both the parameter count and time complexity. By maintaining parameter-free identity shortcuts, deep networks like the 152-layer ResNet achieve higher accuracy than shallower baselines while retaining lower computational complexity (11.3 billion FLOPs) than VGG-16 and VGG-19.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.3 Deep Residual Network Architecture - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Residual Formulation and Shortcut Connections - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Identity versus Projection Shortcuts - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Deeper Bottleneck Architectures - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Dimension Matching and Projection Shortcuts
Bottleneck Residual Blocks
Residual Mapping
Shortcut’s technique for identity mapping
Dimension Matching and Projection Shortcuts
Empirical Performance of Identity vs. Projection Shortcuts
Bottleneck Residual Blocks
Bottleneck Residual Blocks
Deep ResNet Scaling with Bottleneck Blocks
Learn After
What is the primary operational role of the initial convolutional layer within a three-layer bottleneck residual block?
Despite having significantly more layers, ResNet-152 achieves higher accuracy while maintaining lower computational complexity than VGG-16 and VGG-19.
In order from input to output, what are the filter dimensions of the three consecutive convolutions that compose the residual function in a bottleneck block?
Explain why parameter-free identity shortcuts are essential for bottleneck residual architectures, and describe the impact on model efficiency if they are replaced by projection shortcuts.
What is the primary motivation for replacing the standard two-layer residual block with a three-layer bottleneck block in models such as ResNet-50, ResNet-101, and ResNet-152?
Within a three-layer bottleneck residual block, what is the primary operational purpose of the final 1x1 convolution?
In a bottleneck residual block, which points are directly linked by the shortcut connection?
Replacing the parameter-free identity shortcut with a projection shortcut in a bottleneck block doubles both parameter count and time complexity.
What is the computational complexity, measured in FLOPs, of the 152-layer ResNet?
Explain how channel dimensions are managed across the three convolutional layers within a bottleneck residual block, describing the specific role of each convolution in order.