https://towardsdatascience.com/neural-network-architectures-156e5bad51ba

Bottleneck layer

http://datahacker.rs/deep-learning-inception-network/

#017 CNN Inception Network

The bottleneck layer reduces the training time by diminishing the number of features and operations. By reducing the number of nodes of a newer layer in comparison to previous layers, you can reduce dimensionality.

As shown in the figure, the bottom architecture requires 120 M computations, but by adding the bottleneck layer in the middle, in the architecture shown on top, the number of computations is reduced to 12.4 M.

University of Michigan - Ann Arbor

If the input has multiple channels, a 1 x 1 convolution filter would help with combining all the numbers in the corresponding cells of all the input channels into one output number.

If we convolve an *n x n x m* input using *f* channels of 1 x 1 convolution filters, we would get an *n x n x f* output, where each cell is a linear combination (weighted average) of all the corresponding cells in different channels of the input.

This helps with reducing the number of channels to save computational power and memory.

1 x 1 Convolution Layer in Neural Networks  
 (Network ~ Network)

An Inception network (GoogLeNet) is a 22 layer deep neural network that takes many Inception modules (including many convolutional layers) and puts them together, which allow for multiple feature detection.

GoogLeNet is highly accurate, however, because it has so many layers it is more prone to overfitting.

Inception Network (GoogLeNet)

Bottleneck Layer in Inception Network

The top picture is the individual module. The associated node shows a bunch of these modules put together creating the inception network.

Visual of an Inception Module in Inception Network

Auxiliary classifiers are side branches of an inception network take some hidden layers to make a prediction using a few connected layers and Softmax activation to predict the output label.

They help to ensure that the features computed, even at intermediate layers, are not too bad for protecting the output class of a image. In other words, they have some regularizing effect on the inception network and help prevent it from overfitting.

Auxiliary Classifiers in Inception Network

Going deeper with convolutions paper

Using 1x1 convolution to raise and lower the dimension; Convolution repolymerization is performed simultaneously on multiple sizes.

Convolution on multiple scales at the same time can extract the features of different scales, and the features are richer.

The principle of decomposing sparse matrix into dense matrix is used to speed up the convergence speed.

We should bring together highly relevant features. Separate the features of 1x1, 3x3 and 5x5. Because the ultimate goal of training convergence is to extract independent features, gathering highly correlated features in advance can accelerate convergence.

Effect of Inception

1. Inception

2. Multiple loss: perception net has 22 layers deep. In addition to the output of the last layer, it also uses an auxiliary classification node, that is, the output of a middle layer is used as classification and added to the final classification result according to a small weight (0.3), which is equivalent to model fusion. At the same time, it adds back-propagation gradient signal to the network and provides additional regularization.

Learn Before

Related

Learn After