1. ResNeXt
2. Densely Connected CNN
3. Deep Network with Stochastic Depth


Recent Variants of ResNets

- You can make them deeper and deeper without hurting their performance the training set, because all the following layers still get the identity output from previous layers and can ignore the new calculations of previous layers by setting the corresponding weights and biases to zero.
- Their goal is to find the optimal number of layers which reduces the effect of the vanishing gradient problem.

Advantages of ResNets

Plain vs. ResNets Convolutional Neural Network Architectures

- In general, accuracy gains are obtained from increased depth.
-  the 34-layer ResNet has lower error rate than the 18-layer ResNet architecture.
- The 50/101/152-layer ResNets have lower error rate than the 34-layer by considerable margins.

| model | top-1 error (%) |
| ----------- | ----------- |
| ResNet-34 A    | 25.03       |
| ResNet-34 B   | 24.52      |
| ResNet-34 C   | 24.19       |
| ResNet-50  | 22.85      |
| ResNet-101   | 21.75     |
| ResNet-152  | 21.43       |

Evaluate ResNet at different depths for ImageNet Classification

For single model comparison, the 34-layer ResNets was able achieve competitive accuracy, so as other deeper ResNets.

6 ResNet models of different depths with only two 152-layer ones were used to form an ensemble model. It was recorded 3.57% top-5 error on the test set, which was the first place in ILSVRC 2015.

| method     | top-5 err. (test) |
| ----------- | ----------- |
| VGG (ILSVRC’14)    | 7.32%       |
| GoogLeNet   (ILSVRC’14) | 6.66%       |
| VGG (v5)   | 6.8%       |
| PReLU-net    | 4.94%       |
| BN-inception   | 4.82%       |
| ResNet    | 3.57%       |

Evaluate ResNet models with other state-of-the-art models for ImageNet Classification

•	The goal of this technique is to approximate the residual function, instead of stacked layers to approximate the underlying mapping. 

•         Let’s take the residual function F(x): = H(x) – x, where H(x) is the mapping. In this case, the original function thus becomes F(x)+x. 
•       With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings. This method will resolve the degradation problem. 


Residual Learning

•  The "identity shortcuts" are referring to performing the element wise addition of x with the output of the residual layers.

 •  We consider a building block defined as: y = F (x, {Wi}) + x, where x and y are the input and output vectors of the layers considered, the function F (x, {Wi}) represents the residual mapping to be learned.

 •  The residual mapping (F (x, {Wi})) becomes y= W2σ2(W1x)+x
 •  To sum up,  we take the output x of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.



Shortcut’s technique for identity mapping

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

Deep Residual Learning for Image Recognition

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


University of Michigan - Ann Arbor

CNNs are made up of three-layer types—convolutional, pooling and fully-connected (FC).

In the convolutional layers, the input matrix is passed through a set of filters that output a feature map. This output is then sent to a pooling layer, which reduces the size of the feature map. This helps reduce the processing time by condensing the map to its most essential information.

The convolutional and pooling processes are repeated several times, with the number of repeats depending on the network, after which the condensed feature map outputs are sent to a series of FC layers. These FC layers then flatten the maps together and compare the probabilities of each feature occurring in conjunction with the others, until the best classification is determined.

Convolutional Neural Networks Architecture

*Pros:*
- The architecture allows CNNs to learn the position and scale of features in a variety of images, making them especially good at the classification of hierarchical or spatial data and the extraction of unlabeled features.
- Because of **Parameter sharing** and **Sparsity of connections** the neural network has significantly fewer parameters that allows it to train a smaller training sets and be less prone to overfitting.

*Cons:*
- Unfortunately, this structure requires CNNs to only accept fixed-size inputs—and it only allows them to provide fixed-size outputs.

Pros and Cons of CNN Architecture

Fully connected layer is a traditional multilayer perceptron structure. Its input is a one-dimensional vector representing the output of the previous layers. Its output is a list of probabilities for different possible labels attached to the image (e.g. dog, cat, bird). The label that receives the highest probability is the classification decision.

Fully Connected Layer - Classification

Here is a clear 3D representation of the convolution Neural Network for detecting a written number. You can see clearly which layer outputs depend on which inputs:
https://www.cs.ryerson.ca/~aharley/vis/conv/

3D Visualization of a Convolution Neural Network

Three classic networks

A “filter”, sometimes called a “kernel”, is passed over the image, viewing a few pixels at a time (for example, 3X3 or 5X5). The convolution operation is a dot product of the original pixel values with weights defined in the filter. The results are summed up into one number that represents all the pixels the filter observed.

Convolution Filter (Kernel)

In a convolutional layer, instead of manually specifying numbers of cells of the filters (kernels), we define each of them as weights and we add a single bias number to each cell. Then, we train the model with these new parameters the same way as we optimize the weights and biases of neurons. So, the number of parameters in each filter (kernel) will be the number of its cells plus one bias.

The first ConvLayer is responsible for capturing the Low-Level features such as edges, color, gradient orientation, etc. With added layers, the architecture adapts to the high-level features as well, giving us a network which has the wholesome understanding of images in the dataset, similar to how we would.

Convolutional Layer (ConvLayer)

Pooling is the process of down-sampling, i.e., reducing the size of the matrix. A filter is passed over the results of the previous layer and outputs one number out of each group of values.

Pooling Layer in Convolutional Deep Learning

Example of a Convolutional Neural Network Architecture

ResNets Convolutional Neural Network

There are three main classic neural network architectures:
- LeNet-5
- AlexNet
- VGG - 16

Classic Convolutional Neural Network Architectures for Object Detection in Images

An Inception network (GoogLeNet) is a 22 layer deep neural network that takes many Inception modules (including many convolutional layers) and puts them together, which allow for multiple feature detection.

GoogLeNet is highly accurate, however, because it has so many layers it is more prone to overfitting.

Inception Network (GoogLeNet)

The architecture means the overall structure of the network, where most neural networks are organized into groups of units called layers. Different network architecture can have different amount of layers and connections. The main thing to keep in mind when choosing an architecture design, is choosing the depth of the network and the width of each layer.

Architecture Design

We can compare convolution to a fully connected net, but with an inﬁnitely strong prior over its weights: 
                                                                                                                                                                           
1. the weights for one hidden unit must be identical to the weights of its neighbor but shifted in space. 
2. the weights must be zero, except for the small, spatially contiguous receptive ﬁeld assigned to that hidden unit.
                                                                                                                                                                           
Pooling is similar:  
                                                         
1. contains only local interactions and is equivariant to translation
2. each unit should be invariant to small translations.


Learn Before

Related

Learn After