Plain versus Residual Baseline Architectures
The baseline plain architectures for ImageNet are primarily inspired by VGG nets. Their convolutional layers mostly use filters following two core design rules: layers with the same output feature map size have the same number of filters, and when the feature map size is halved, the filter count is doubled to preserve time complexity per layer. Downsampling is performed directly via convolutional layers with a stride of 2. The network terminates with a global average pooling layer followed by a 1000-way fully-connected layer with softmax. A 34-layer plain baseline constructed this way requires 3.6 billion FLOPs, representing only 18% of VGG-19's 19.6 billion FLOPs.
Evaluating plain networks reveals the degradation problem: the 34-layer plain network achieves a higher validation error (28.54% top-1) than the shallower 18-layer plain network (27.94% top-1). This optimization difficulty is not caused by vanishing gradients, as the networks are trained with Batch Normalization (BN) to guarantee non-zero forward signal variances and healthy backward gradient norms.
In counterpart residual networks (ResNets), shortcut connections are inserted across pairs of filters. With identity shortcuts and zero-padding for dimension matching, the degradation phenomenon is completely reversed: ResNet-34 achieves a 25.03% top-1 error, outperforming ResNet-18 (27.88%) by 2.8% and its plain counterpart by 3.5%. ResNet-34 demonstrates noticeably lower training error throughout optimization, proving that residual formulations enable deeper networks to gain accuracy from added depth.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.4 Residual Network Experiments and Applications - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
ImageNet Classification and Model Variations - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
In the plain baseline architecture inspired by VGG, what rule is applied to the filter count when the feature map size is halved?
In plain baseline architectures, the degradation problem observed when increasing network depth is caused by vanishing gradients.
How does the plain baseline network perform downsampling, and what layer does it terminate with immediately before the 1000-way fully-connected layer?
Compare the performance of plain networks and residual networks when scaling depth from 18 to 34 layers on ImageNet. In your response, describe the degradation problem observed in the plain baselines, how residual networks resolve this issue, and what training behavior demonstrates that residual networks effectively benefit from added depth.