Good article for buliding an intuition on how to derive the backpropagation formula:https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c

University of California, Berkeley

A “filter”, sometimes called a “kernel”, is passed over the image, viewing a few pixels at a time (for example, 3X3 or 5X5). The convolution operation is a dot product of the original pixel values with weights defined in the filter. The results are summed up into one number that represents all the pixels the filter observed.

Convolution Filter (Kernel)

Convolutions and Backpropagations

The back-propagation in CNN is done on the same principle of the chain rule, but the the the way it is actually done is not that easy. Of course if we have the gradient of the output we could do bunch of nested loops and calculate the gradients of the parameters and inputs, but this would be crazy inefficient. We need to search a way to vectorize this. We already implemented the forward propagation, and convolution operation is heavily involved in back propagation. The formulas for the gradients of the input are the following:(Suppose L is the cost function, F is the current filter, X is the current input)


Backpropagation in CNN

- The convolution operation, denoted by the asterisk (*), indicates mapping the filter matrix (kernel) to every possible position on the input matrix, multiplying each of the pairs of numbers, elementwise, and adding them up to calculate the corresponding element in the result matrix.


- In mathematics, the asterisk is the standard symbol for convolution but in Python, this is also used to denote multiplication or elementwise multiplication.


- Traditionally, the convolution operation before its application has a kernel mirroring step (left to right, and up to down). This allows associativity across convolutions to hold. In deep learning, this mirroring doesn't have a significant impact on the operations and thus is omitted. Also when the kernel is symmetric, the operations are identical. Hence, convolution in deep learning refers to cross-correlation. 

The Cross-Correlation (Convolution) Operation

The size of one dimension of the filter (kernel) matrix. Because it's usually a square matrix, we only consider the size of one dimension which is usually the same as the other dimension.

In machine vision, it is usually an **odd** number. So, we usually have a center cell in the filter (kernel) matrix.

Filter (Kernel) Size in Convolutional DNN

https://medium.com/@RosieCampbell/demystifying-deep-neural-nets-efb726eae941

Demystifying Deep Neural Nets

In a convolutional layer, instead of manually specifying numbers of cells of the filters (kernels), we define each of them as weights and we add a single bias number to each cell. Then, we train the model with these new parameters the same way as we optimize the weights and biases of neurons. So, the number of parameters in each filter (kernel) will be the number of its cells plus one bias.

The first ConvLayer is responsible for capturing the Low-Level features such as edges, color, gradient orientation, etc. With added layers, the architecture adapts to the high-level features as well, giving us a network which has the wholesome understanding of images in the dataset, similar to how we would.

Convolutional Layer (ConvLayer)

Three basic strategies for obtaining convolution kernels without supervised training:
- Initializing them randomly
- Setting each kernel to detect edges at a certain orientation or scale
- Item learning the kernels with an unsupervised criterion

Learn Before

Related