The introduction of residual connections had a major influence on the design of subsequent deep neural networks, spanning both convolutional and sequential domains. A prominent example is the Transformer architecture, which adopts residual connections alongside other design choices. This combination has become pervasive across diverse fields, including natural language processing, computer vision, speech recognition, and reinforcement learning.

Claude

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


ResNets Convolutional Neural Network

Dive into Deep Learning

1. ResNeXt
2. Densely Connected CNN
3. Deep Network with Stochastic Depth


Recent Variants of ResNets

- You can make them deeper and deeper without hurting their performance the training set, because all the following layers still get the identity output from previous layers and can ignore the new calculations of previous layers by setting the corresponding weights and biases to zero.
- Their goal is to find the optimal number of layers which reduces the effect of the vanishing gradient problem.

Advantages of ResNets

Plain vs. ResNets Convolutional Neural Network Architectures

- In general, accuracy gains are obtained from increased depth.
-  the 34-layer ResNet has lower error rate than the 18-layer ResNet architecture.
- The 50/101/152-layer ResNets have lower error rate than the 34-layer by considerable margins.

| model | top-1 error (%) |
| ----------- | ----------- |
| ResNet-34 A    | 25.03       |
| ResNet-34 B   | 24.52      |
| ResNet-34 C   | 24.19       |
| ResNet-50  | 22.85      |
| ResNet-101   | 21.75     |
| ResNet-152  | 21.43       |

Evaluate ResNet at different depths for ImageNet Classification

For single model comparison, the 34-layer ResNets was able achieve competitive accuracy, so as other deeper ResNets.

6 ResNet models of different depths with only two 152-layer ones were used to form an ensemble model. It was recorded 3.57% top-5 error on the test set, which was the first place in ILSVRC 2015.

| method     | top-5 err. (test) |
| ----------- | ----------- |
| VGG (ILSVRC’14)    | 7.32%       |
| GoogLeNet   (ILSVRC’14) | 6.66%       |
| VGG (v5)   | 6.8%       |
| PReLU-net    | 4.94%       |
| BN-inception   | 4.82%       |
| ResNet    | 3.57%       |

Evaluate ResNet models with other state-of-the-art models for ImageNet Classification

•  The "identity shortcuts" are referring to performing the element wise addition of x with the output of the residual layers.

 •  We consider a building block defined as: y = F (x, {Wi}) + x, where x and y are the input and output vectors of the layers considered, the function F (x, {Wi}) represents the residual mapping to be learned.

 •  The residual mapping (F (x, {Wi})) becomes y= W2σ2(W1x)+x
 •  To sum up,  we take the output x of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.



Shortcut’s technique for identity mapping

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778). https://doi.org/10.1109/cvpr.2016.90

Deep Residual Learning for Image Recognition

In a residual network, the desired underlying mapping $$f(\mathbf{x})$$—the function the network ultimately aims to approximate—is not learned directly by a stack of layers. Instead, those layers are reformulated to learn only the residual mapping $$g(\mathbf{x}) = f(\mathbf{x}) - \mathbf{x}$$, and the target function is recovered as $$f(\mathbf{x}) = g(\mathbf{x}) + \mathbf{x}$$. This reformulation is motivated by the degradation problem: as plain networks grow deeper, their training accuracy can paradoxically worsen, suggesting that the added layers struggle to approximate even the identity function. By recasting the problem in terms of $$g(\mathbf{x})$$, the identity case $$f(\mathbf{x}) = \mathbf{x}$$ reduces to $$g(\mathbf{x}) = 0$$, which is significantly easier for a network to learn because it only requires driving the weights and biases of the constituent layers toward zero.

Residual Mapping

In the original ResNet architecture designed for ImageNet-scale images, the initial layers consist of a $$7 	imes 7$$ convolutional layer with $$64$$ output channels and a stride of $$2$$, followed by batch normalization, a ReLU activation, and a $$3 	imes 3$$ max-pooling layer with a stride of $$2$$. These large receptive fields and aggressive downsampling are appropriate for the $$224 	imes 224$$ pixel inputs common in ImageNet. However, when working with significantly smaller images (e.g., $$28 	imes 28$$ or $$96 	imes 96$$ pixels from datasets like Fashion-MNIST or CIFAR), these initial layers would reduce spatial dimensions too aggressively, leaving insufficient resolution for the subsequent residual blocks to extract meaningful features.

ResNet Initial Layers

Prior to the widespread adoption of residual connections, highway networks were introduced to enable the effective training of deep architectures with over 100 layers. Highway networks achieved this by using bypassing paths controlled by parameterized gating units. In contrast, Residual Networks (ResNets) simplified this approach by using parameter-free identity functions as their bypassing paths, a design choice that performed remarkably well on various computer vision tasks.

Highway Networks vs. Residual Networks

Influence of Residual Connections on Subsequent Architectures

A significant practical advantage of residual networks is the ability to dynamically add layers during the training process. Because the default behavior of a residual block is to let data pass through unchanged via the identity mapping, new layers initialized as identity functions can be inserted without disrupting the already learned representations. This technique can actively accelerate the training phase of very large neural networks.

Adding Layers During Training in Residual Networks

Residual connections allow inputs to forward propagate faster across layers by providing a direct path that bypasses intermediate transformations. This accelerated propagation of information facilitates the training of significantly deeper neural networks without degradation. As a consequence, architectures can scale to extensive depths, such as the $$152$$-layer model introduced in the original ResNet research.

Learn Before

Related