- This is a 34 layer network.
- 3x3 filters are deployed in most of the convolutional layers in this model.
- Dimensionality reduction of feature map is achieved using a stride of 2 instead of 
   maxpool layer as in VGG-19
- Feature map's dimensionality reduction results in doubling of filters used in order to 
   conserve time complexity
- The number of filters in this network is less than that of VGG-19.
- This model has only 18% floating point operation of VGG-19 network.

Plain Network for ImageNet Dataset

- Short connections are inserted in the plain 34-layer network.
- Solid lines showcases connections having same dimension.
- Dotted lines indicate connections having varying dimensions.
- Either zero padding or projection shortcut to increase dimension.

ResNet  for ImageNet Dataset

- Image resizing for sale augmentation
- Batch normalisation after convolution and before activation.
- Stochastic gradient descent with a mini batch size of 256

Implementation of Networks

1. 34-layer ResNet reduces top-1 error by 3.5%, compared to 34-layer plain convolutional neural network:

2. 18-layer ResNet convereges faster than 18-layer plain convolutional neural network in early stage.

|     | plain | ResNet |
| -----------| ----------- | ----------- |
| 18-layer | 27.94 %     | 27.88%       |
| 34-layer | 28.54%   | 25.03%      |


Plain vs. ResNets Convolutional Neural Network Model Evaluation for ImageNet Classification

University of Michigan - Ann Arbor

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


ResNets Convolutional Neural Network

1. ResNeXt
2. Densely Connected CNN
3. Deep Network with Stochastic Depth


Recent Variants of ResNets

- You can make them deeper and deeper without hurting their performance the training set, because all the following layers still get the identity output from previous layers and can ignore the new calculations of previous layers by setting the corresponding weights and biases to zero.
- Their goal is to find the optimal number of layers which reduces the effect of the vanishing gradient problem.

Advantages of ResNets

Plain vs. ResNets Convolutional Neural Network Architectures

- In general, accuracy gains are obtained from increased depth.
-  the 34-layer ResNet has lower error rate than the 18-layer ResNet architecture.
- The 50/101/152-layer ResNets have lower error rate than the 34-layer by considerable margins.

| model | top-1 error (%) |
| ----------- | ----------- |
| ResNet-34 A    | 25.03       |
| ResNet-34 B   | 24.52      |
| ResNet-34 C   | 24.19       |
| ResNet-50  | 22.85      |
| ResNet-101   | 21.75     |
| ResNet-152  | 21.43       |

Evaluate ResNet at different depths for ImageNet Classification

For single model comparison, the 34-layer ResNets was able achieve competitive accuracy, so as other deeper ResNets.

6 ResNet models of different depths with only two 152-layer ones were used to form an ensemble model. It was recorded 3.57% top-5 error on the test set, which was the first place in ILSVRC 2015.

| method     | top-5 err. (test) |
| ----------- | ----------- |
| VGG (ILSVRC’14)    | 7.32%       |
| GoogLeNet   (ILSVRC’14) | 6.66%       |
| VGG (v5)   | 6.8%       |
| PReLU-net    | 4.94%       |
| BN-inception   | 4.82%       |
| ResNet    | 3.57%       |

Evaluate ResNet models with other state-of-the-art models for ImageNet Classification

•	The goal of this technique is to approximate the residual function, instead of stacked layers to approximate the underlying mapping. 

•         Let’s take the residual function F(x): = H(x) – x, where H(x) is the mapping. In this case, the original function thus becomes F(x)+x. 
•       With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings. This method will resolve the degradation problem. 


Residual Learning

•  The "identity shortcuts" are referring to performing the element wise addition of x with the output of the residual layers.

 •  We consider a building block defined as: y = F (x, {Wi}) + x, where x and y are the input and output vectors of the layers considered, the function F (x, {Wi}) represents the residual mapping to be learned.

 •  The residual mapping (F (x, {Wi})) becomes y= W2σ2(W1x)+x
 •  To sum up,  we take the output x of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.



Shortcut’s technique for identity mapping

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

Learn Before

Related

Learn After