•  The "identity shortcuts" are referring to performing the element wise addition of x with the output of the residual layers.

 •  We consider a building block defined as: y = F (x, {Wi}) + x, where x and y are the input and output vectors of the layers considered, the function F (x, {Wi}) represents the residual mapping to be learned.

 •  The residual mapping (F (x, {Wi})) becomes y= W2σ2(W1x)+x
 •  To sum up,  we take the output x of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.



University of Pittsburgh

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


ResNets Convolutional Neural Network

1. ResNeXt
2. Densely Connected CNN
3. Deep Network with Stochastic Depth


Recent Variants of ResNets

- You can make them deeper and deeper without hurting their performance the training set, because all the following layers still get the identity output from previous layers and can ignore the new calculations of previous layers by setting the corresponding weights and biases to zero.
- Their goal is to find the optimal number of layers which reduces the effect of the vanishing gradient problem.

Advantages of ResNets

Plain vs. ResNets Convolutional Neural Network Architectures

- In general, accuracy gains are obtained from increased depth.
-  the 34-layer ResNet has lower error rate than the 18-layer ResNet architecture.
- The 50/101/152-layer ResNets have lower error rate than the 34-layer by considerable margins.

| model | top-1 error (%) |
| ----------- | ----------- |
| ResNet-34 A    | 25.03       |
| ResNet-34 B   | 24.52      |
| ResNet-34 C   | 24.19       |
| ResNet-50  | 22.85      |
| ResNet-101   | 21.75     |
| ResNet-152  | 21.43       |

Evaluate ResNet at different depths for ImageNet Classification

For single model comparison, the 34-layer ResNets was able achieve competitive accuracy, so as other deeper ResNets.

6 ResNet models of different depths with only two 152-layer ones were used to form an ensemble model. It was recorded 3.57% top-5 error on the test set, which was the first place in ILSVRC 2015.

| method     | top-5 err. (test) |
| ----------- | ----------- |
| VGG (ILSVRC’14)    | 7.32%       |
| GoogLeNet   (ILSVRC’14) | 6.66%       |
| VGG (v5)   | 6.8%       |
| PReLU-net    | 4.94%       |
| BN-inception   | 4.82%       |
| ResNet    | 3.57%       |

Evaluate ResNet models with other state-of-the-art models for ImageNet Classification

•	The goal of this technique is to approximate the residual function, instead of stacked layers to approximate the underlying mapping. 

•         Let’s take the residual function F(x): = H(x) – x, where H(x) is the mapping. In this case, the original function thus becomes F(x)+x. 
•       With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings. This method will resolve the degradation problem. 


Residual Learning

Shortcut’s technique for identity mapping

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

Learn Before

Related