The convolution should be equivariant to the translation function g(x), i.e., a function that changes the input of the function f(x). So in a convolution layer, if a feature in the image gets moved, the kernel will still go over it and will detect its representation.

University of Michigan - Ann Arbor

University of California, Berkeley

*Pros:*
- The architecture allows CNNs to learn the position and scale of features in a variety of images, making them especially good at the classification of hierarchical or spatial data and the extraction of unlabeled features.
- Because of **Parameter sharing** and **Sparsity of connections** the neural network has significantly fewer parameters that allows it to train a smaller training sets and be less prone to overfitting.

*Cons:*
- Unfortunately, this structure requires CNNs to only accept fixed-size inputs—and it only allows them to provide fixed-size outputs.

Pros and Cons of CNN Architecture

We denote f(x) to be equivariant to g(x) if f(g(x)) = g(f(x)).

Equivariant Functions

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

In a Feed-Forward NN, each neuron has its own set of parameters that affect it and different neurons have distinct set of parameters. Each parameter is used exactly once.

In contrast, in CNNs using the kernel convolution means that this kernel affects several output elements at once. This is because a feature detector (such as a vertical edge detector) that is useful in one part of the image, is probably useful in another part of the image. The main advantage is that it decreases the memory requirement significantly.

As you can see in the example arrows of the same color indicate the same parameters.

Parameter Sharing in Convolutional Neural Networks

In a Feed-forward NN, we use the linear matrix multiplication for the forward propagation and each element in the output is dependent on each element in the input. Now imagine the input is a 500 x 500 image - including the three image channels, we have the input of 3 x 500 x 500 = 750,000 dimensions. Considering this number as only one side of the parameter matrix in the first layer, this means a lot of parameters in the first layer.

In contrast, in a convolution layer, each output value only depends on a small number of inputs, allowing us to significantly improve on efficiency, and decrease memory requirements. This way, we will be able to use very large input images.

Learn Before

Related