For a convolutional layer $$l$$, the output dimensions and number of parameters are calculated as follows:

Input Size: $$n_h^{[l-1]} 	imes n_w^{[l-1]} 	imes n_c^{[l-1]}$$
Filter (Kernel) Size: $$k_h^{[l]} 	imes k_w^{[l]} 	imes n_c^{[l-1]}$$
*(Note: $$n_c^{[l]}$$ is the number of filters in layer $$l$$, matching the number of output channels.)
Stride: $$s_h^{[l]} 	imes s_w^{[l]}$$
Padding: $$p_h^{[l]} 	imes p_w^{[l]}$$

Output (Activations $$a^{[l]}$$) Size: $$n_h^{[l]} 	imes n_w^{[l]} 	imes n_c^{[l]}$$
$$n_h^{[l]} = \left \lfloor \frac{n_h^{[l-1]} + 2p_h^{[l]} - k_h^{[l]}}{s_h^{[l]}} ight floor + 1$$
$$n_w^{[l]} = \left \lfloor \frac{n_w^{[l-1]} + 2p_w^{[l]} - k_w^{[l]}}{s_w^{[l]}} ight floor + 1$$

Number of Parameters:
 Weights: $$k_h^{[l]} 	imes k_w^{[l]} 	imes n_c^{[l-1]} 	imes n_c^{[l]}$$
* Biases: $$n_c^{[l]}$$

Convolution Layer Output Size and Parameter Formulas

While it is possible to manually design simple convolution kernels for specific tasks—such as using a $$1 \times 2$$ finite difference operator like $$[1, -1]$$ for basic edge detection—this approach does not scale to complex problems. As deep learning models utilize larger kernels and incorporate successive layers of convolutions, it becomes practically impossible to manually specify the precise weights and the specific function of every individual filter.

Limitation of Manually Designed Convolutional Kernels

In deep learning, operations labeled as convolutions are technically cross-correlations. A strict two-dimensional discrete convolution uses index differences to sum elements, effectively flipping the kernel: $$(f * g)(i, j) = \sum_a \sum_b f(a, b) g(i-a, j-b)$$. In contrast, cross-correlation uses index additions: $$(i+a, j+b)$$. Because kernels are learned from data, a network using cross-correlation will simply learn a flipped version of the kernel matrix $$\mathbf{K}'$$ that a strict convolution would learn as $$\mathbf{K}$$, producing identical outputs. This cosmetic distinction justifies the standard terminology of calling cross-correlation a convolution.

Equivalence of Strict Convolution and Cross-Correlation

In the context of convolutional neural networks, a feature map is the output produced by a convolutional layer. It acts as a learned representation, capturing specific spatial features—such as patterns across height and width dimensions—that are then passed forward as input to subsequent layers in the network.

Feature Map

A two-dimensional convolutional layer can be implemented programmatically by defining a class that encapsulates its two core parameters: the kernel weights and the scalar bias. In the class constructor, these parameters are declared and initialized (e.g., initializing weights randomly and bias to zero). The layer's forward propagation method then computes the output by executing a two-dimensional cross-correlation between the input and the kernel weights, and subsequently adding the scalar bias to the result.

Two-Dimensional Convolutional Layer Code Implementation

The dimensions of a convolution kernel are defined by its height ($$h$$) and width ($$w$$), commonly referred to as an $$h 	imes w$$ convolution kernel. Accordingly, the operation utilizing this kernel is known as an $$h 	imes w$$ convolution. Furthermore, a convolutional layer that employs an $$h 	imes w$$ convolution kernel is simply denoted as an $$h 	imes w$$ convolutional layer.

Convolution Kernel and Layer Size Notation

A practical illustration of a convolutional layer is a Waldo detector, which scans across an image to locate the character Waldo. The layer extracts small, localized windows from the input image and weights the pixel intensities using a specifically trained convolution filter, denoted as $$\mathsf{V}$$. The objective is to learn a filter such that locations in the image with the highest 'waldoness'—meaning they closely match the target pattern—result in peak activation values in the corresponding spatial locations of the hidden representation.

Waldo Detector Convolution Example

Edge detection is a fundamental image processing application of convolutional layers. It involves locating the boundaries of an object within an image by finding the precise locations of pixel changes. By performing a cross-correlation or convolution operation with a specifically constructed filter matrix, or kernel, the layer can successfully detect distinct kinds of boundaries, such as vertical or horizontal edges.

Object Edge Detection Using Convolution

The fundamental computation of a convolutional layer—typically implemented as a cross-correlation operation—is highly local, meaning that each output element depends only on a small, contiguous region of the input. This structural locality allows for significant hardware optimization, as chip designers can prioritize fast computation units over large memory capacity or bandwidth when designing accelerators for convolutions. While this specialized hardware design might not be optimal for all types of algorithms, it provides the computational efficiency necessary for affordable and ubiquitous computer vision applications.

Locality and Hardware Optimization of Convolutions

In modern convolutional neural network architectures, it is essential to utilize multiple channels at each layer. As data propagates deeper into the network, the channel dimension is typically increased. This architectural design explicitly trades off spatial resolution, which is usually downsampled, for greater channel depth, allowing the network to capture increasingly complex and jointly optimized feature representations.

Channel Depth in Convolutional Networks

A convolutional layer performs a cross-correlation operation between its input and a kernel, and subsequently adds a scalar bias to the result to generate an output. During model training, the convolutional layer learns two key parameters: the kernel (weights) and the scalar bias. Prior to training, these kernel weights are generally initialized with random values, analogous to the initialization process in fully connected layers.

University of Michigan - Ann Arbor

Claude

CNNs are made up of three-layer types—convolutional, pooling and fully-connected (FC).

In the convolutional layers, the input matrix is passed through a set of filters that output a feature map. This output is then sent to a pooling layer, which reduces the size of the feature map. This helps reduce the processing time by condensing the map to its most essential information.

The convolutional and pooling processes are repeated several times, with the number of repeats depending on the network, after which the condensed feature map outputs are sent to a series of FC layers. These FC layers then flatten the maps together and compare the probabilities of each feature occurring in conjunction with the others, until the best classification is determined.

Convolutional Neural Networks Architecture

A “filter”, sometimes called a “kernel”, is passed over the image, viewing a few pixels at a time (for example, 3X3 or 5X5). The convolution operation is a dot product of the original pixel values with weights defined in the filter. The results are summed up into one number that represents all the pixels the filter observed.

Convolution Filter (Kernel)

The cross-correlation operation involves mapping a filter matrix (or kernel) to every possible position on an input matrix. At each position, the corresponding elements of the input and the kernel are multiplied together, and the products are summed to calculate a single entry in the resulting output matrix. While a strict mathematical convolution requires flipping the two-dimensional kernel both horizontally and vertically prior to this process, deep learning implementations typically omit this mirroring step and compute the cross-correlation directly. Despite this technical distinction, it is standard terminology in deep learning literature to refer to the cross-correlation operation simply as a convolution.

The Cross-Correlation (Convolution) Operation

https://medium.com/@RosieCampbell/demystifying-deep-neural-nets-efb726eae941

Demystifying Deep Neural Nets

Dive into Deep Learning

*Pros:*
- The architecture allows CNNs to learn the position and scale of features in a variety of images, making them especially good at the classification of hierarchical or spatial data and the extraction of unlabeled features.
- Because of **Parameter sharing** and **Sparsity of connections** the neural network has significantly fewer parameters that allows it to train a smaller training sets and be less prone to overfitting.

*Cons:*
- Unfortunately, this structure requires CNNs to only accept fixed-size inputs—and it only allows them to provide fixed-size outputs.

Pros and Cons of CNN Architecture

Fully connected layer is a traditional multilayer perceptron structure. Its input is a one-dimensional vector representing the output of the previous layers. Its output is a list of probabilities for different possible labels attached to the image (e.g. dog, cat, bird). The label that receives the highest probability is the classification decision.

Fully Connected Layer - Classification

Here is a clear 3D representation of the convolution Neural Network for detecting a written number. You can see clearly which layer outputs depend on which inputs:
https://www.cs.ryerson.ca/~aharley/vis/conv/

3D Visualization of a Convolution Neural Network

Three classic networks

Pooling operators use a fixed-shape window that slides over all regions of the input tensor based on a specified stride. For each location traversed, this pooling window computes a single output. Unlike convolutional layers that compute cross-correlations using learnable parameters (kernels), the pooling layer contains no parameters. Instead, pooling operations are deterministic, typically calculating the maximum or average value within the pooling window.

Pooling Layer in Convolutional Deep Learning

Example of a Convolutional Neural Network Architecture

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


ResNets Convolutional Neural Network

There are three main classic neural network architectures:
- LeNet-5
- AlexNet
- VGG - 16

Classic Convolutional Neural Network Architectures for Object Detection in Images

The Inception network, also known as GoogLeNet, is a deep convolutional neural network architecture that arranges multiple Inception modules (multi-branch convolutional blocks) into a sequential pipeline for multi-scale feature detection. GoogLeNet uses a stack of $$9$$ Inception blocks organized into three groups with max-pooling between them, and employs global average pooling in its output head to generate predictions. The stem of the network resembles earlier architectures like AlexNet and LeNet. Max-pooling between Inception block groups reduces the spatial dimensionality. The model is computationally complex and involves a large number of relatively arbitrary hyperparameters governing channel counts, the number of blocks before dimensionality reduction, and the relative partitioning of capacity across channels.

Inception Network (GoogLeNet)

The architecture means the overall structure of the network, where most neural networks are organized into groups of units called layers. Different network architecture can have different amount of layers and connections. The main thing to keep in mind when choosing an architecture design, is choosing the depth of the network and the width of each layer.

Architecture Design

We can compare convolution to a fully connected net, but with an inﬁnitely strong prior over its weights: 
                                                                                                                                                                           
1. the weights for one hidden unit must be identical to the weights of its neighbor but shifted in space. 
2. the weights must be zero, except for the small, spatially contiguous receptive ﬁeld assigned to that hidden unit.
                                                                                                                                                                           
Pooling is similar:  
                                                         
1. contains only local interactions and is equivariant to translation
2. each unit should be invariant to small translations.


Convolution and Pooling as an Inﬁnitely Strong Prior

Convolutional Layer

Although convolutional neural networks typically contain fewer learnable parameters than comparably deep multi-layer perceptrons, they can still be more computationally expensive to train. The reason is parameter sharing: each convolutional kernel weight participates in many more multiplications because it is applied across every spatial position in the input feature map. As a result, the total number of floating-point operations can exceed that of a similarly structured MLP, even though the CNN stores far fewer unique weights. This makes GPU acceleration especially valuable when training CNNs, since the parallelism of GPUs is well-suited to the large number of repeated multiply–accumulate operations that convolutions require.

CNN Computational Cost versus Parameter Count

In modern deep learning models, particularly Convolutional Neural Networks (CNNs), feature representations are learned automatically rather than being manually engineered. These features are composed hierarchically across multiple jointly learned layers. The lowest layers of the network typically learn to extract basic visual elements, such as edges, colors, and simple textures. Intermediate layers build upon these foundational features to detect more complex structures like object parts, while the highest layers assemble them to recognize entire objects. The final hidden state provides a compact representation that summarizes the image contents, allowing data belonging to different categories to be easily separated by a classifier.

Hierarchical Feature Learning in Vision Networks

The standard building block of early Convolutional Neural Networks consists of a convolutional layer with padding to maintain resolution, a nonlinearity (such as a ReLU), and a pooling layer (such as max-pooling) that reduces the spatial resolution. Because max-pooling typically halves the resolution in each block, this approach imposes a strict limit on network depth. Specifically, a network with an input dimension $$d$$ can accommodate at most $$\log_2 d$$ such convolutional layers before the spatial dimensions are entirely exhausted. For example, on ImageNet data, this architectural design restricts the network to a maximum of $$8$$ convolutional layers.

Spatial Resolution Limit in CNNs

ParNet is a neural network architecture that achieves competitive performance using a much shallower design compared to deep convolutional models like VGG. Instead of relying on a deep, sequential progression of layers, ParNet utilizes a large number of parallel computations to process information. This approach demonstrates a viable alternative to the dominant trend of increasingly deep networks, suggesting that highly parallelized shallow architectures can also be highly effective for modern deep learning tasks.

ParNet Architecture

The Network in Network (NiN) architecture introduces the concept of applying a fully connected layer at each individual pixel location (height and width) of a feature map. Convolutional layers output four-dimensional tensors corresponding to the example, channel, height, and width, whereas traditional fully connected layers expect two-dimensional tensors corresponding to the example and feature. To bridge this gap without flattening the spatial dimensions prematurely, NiN utilizes $$1 	imes 1$$ convolutions. A $$1 	imes 1$$ convolution effectively acts as a fully connected layer operating independently on each pixel's channels across the spatial grid.

Network in Network (NiN) Architecture

ShiftNet is a convolutional neural network architecture designed to increase functional complexity without adding computational cost. It achieves this by mimicking the effects of a standard $$3 \times 3$$ spatial convolution through a simpler mechanism: it explicitly adds spatially shifted activations across different channels. This "shift" operation requires zero floating-point operations and zero parameters, serving as a highly efficient alternative to traditional convolutions.

ShiftNet Architecture

Traditionally, the design of deep neural networks has been a highly manual process, relying heavily on human intuition to find optimal hyperparameters and structural configurations. Because this manual approach is costly in terms of human time and does not guarantee optimal outcomes, researchers developed the notion of network design spaces. Instead of manually designing a single specific network instance, this automated strategy parameterizes a population of possible architectures—a design space—and searches within it to systematically discover high-quality models, such as the RegNetX and RegNetY families.

Network Design Spaces

Good article for buliding an intuition on how to derive the backpropagation formula:https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c

Convolutions and Backpropagations

The back-propagation in CNN is done on the same principle of the chain rule, but the the the way it is actually done is not that easy. Of course if we have the gradient of the output we could do bunch of nested loops and calculate the gradients of the parameters and inputs, but this would be crazy inefficient. We need to search a way to vectorize this. We already implemented the forward propagation, and convolution operation is heavily involved in back propagation. The formulas for the gradients of the input are the following:(Suppose L is the cost function, F is the current filter, X is the current input)


Backpropagation in CNN

Three basic strategies for obtaining convolution kernels without supervised training:
- Initializing them randomly
- Setting each kernel to detect edges at a certain orientation or scale
- Item learning the kernels with an unsupervised criterion

Obtaining Kernels without Supervised Training

This node before explained how it works, and now we need to understand how to write the operation mathematically and more importantly code it efficiently. So suppose we have the input I with the shape H x H x C, where B - the batch, H - is the height and width of the image( just taking square image for the example), C - the amount of channels. One kernel k has a size of K x K x C( also taking square kernel for the sake of the example). b is the bias for that kernel

$ ( I * K)_{i,j} = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} \sum_{c=0}^{C} K_{m,n,c} I_{i + m, j + n, c} + b$
-------------------------------------------------
$ ( I * K)$ - is the output matrix for the current kernel with size of H-K+1, H-K+1. This is the variation of the Convolution with no padding and stride one. Most of the libraries call it convolution but it is actually an operation called cross-correlation. In terms of coding the formula for convolution, there are several ways to implement it. One of them might be just straight brute force with nested loops, turning everything int a big matrix multiplication, and even using Fourier transforms. While brute force solution is straight forward, other two ways are not that easy to visualize and understand


Mathematical Implementation of Forward Propagation

Here is the visualization of the convolution operation were you can play with different hyperparameters of CNN:
https://ezyang.github.io/convolution-visualizer/index.html

Convolution Visualizer

On the left, we have a 6 x 6 grayscale image, so we represent it using a 6 x 6 x 1 matrix because there are no separate RGB channels. To detect vertical edges, we construct a 3 x 3 matrix, called filter or kernel, shown in the middle. We convolve the 6 x 6 matrix with the 3 x 3 filter. The output of this convolution operator will be a 4 x 4 matrix.

To compute the upper left element of this 4 x 4 matrix, we take the 3 x 3 filter and paste it on top of the 3 x 3 region of the original input image and sum up all the 9 the element-wise products. So, the result would be 3 x 1 + 1 x 1 + 2 x 1 + 0 x 0 + 5 x 0 + 7 x 0 + 1 x (-1) + 8 x (-1) + 2 x (-1) = -5. Next, to compute the second element, we shift the pasted filter matrix one step to the right on the 6 x 6 image, do the same element-wise products, and then add them up, which will give us -4, and so on.

To compute the next rows, we shift the pasted filter matrix one step down on the 6 x 6 image. Then, we repeat the same element-wise products, and additions for the second row, and so on.

Calculating Cross-Correlation (Convolution) Operation Example

In a standard cross-correlation operation, the convolution window slides over the input tensor one element at a time. However, to increase computational efficiency or to downsample the representation, we can traverse multiple elements per slide, skipping intermediate locations. The number of rows and columns traversed per slide is called the stride. Strided convolution is particularly useful when the convolution kernel is large, as it efficiently captures a broad area of the underlying image.

Strided Convolution

A 2D convolution layer performs an entry per entry multiplication between the input and the filter, where the input and the filter are 2D matrices. 

In a 3D convolution layer, the same operation is performed on 3D inputs and the filters, such that:
- We multiply corresponding entries in the input and filter cubes and add up the results to determine the output entry.
- In addition to shifting (striding) the filter on two dimensions of the input matrix to perform the convolution operation, we also shift it in the third dimension.

For example, if we have a 4 x 4 x 4 input and a 3 x 3 x 3 filter, because we can shift the filter for only one entry per dimension, the output will be 2 x 2 x 2.

3D Convolution Layers

For example, if we have a 3 x 3 x 3 RGB filter, to perform the convolution operation, we start with placing the filter in the upper left most position of the image. We can consider this 3 x 3 x 3 filter as a cube with 27 parameters, where the first dimension is called "width," the second is called height, and the third indicates the number of channels. We multiply each of these 27 numbers with the corresponding numbers from the red, green, and blue channels of the image and sum up the results to calculate the corresponding output number. Then to compute the next output, we take this cube and slide it over by one, do the 27 multiplications, and add up the results to calculate the next output number, and so on. This way, convolving a 6 x 6 x 3 image with a 3 x 3 x 3 filter will give us a 4 x 4 output.

Convolutions Over Volumes

To execute a two-dimensional cross-correlation, the convolution window starts at the upper-left of the input tensor and systematically translates across it, moving rightwards and downwards. At each valid position, the kernel tensor undergoes an elementwise multiplication with the currently overlaid input subtensor. The sum of these individual products produces a single scalar, which becomes the corresponding element in the resulting output tensor.

Two-Dimensional Convolution Operation Procedure

When applying a two-dimensional cross-correlation without padding, the resulting output tensor is dimensionally smaller than the input. Since the kernel must fit completely within the boundaries of the input tensor, an input of size $$n_\textrm{h} \times n_\textrm{w}$$ convolved with a kernel of size $$k_\textrm{h} \times k_\textrm{w}$$ will yield an output shape calculated by the formula: $$(n_\textrm{h}-k_\textrm{h}+1) \times (n_\textrm{w}-k_\textrm{w}+1)$$.

Computation of Convolution Output Size

A convolution kernel can act as a finite difference operator to locate pixel changes for edge detection. For example, a $$1 \times 2$$ kernel like $$[1, -1]$$ computes the difference between horizontally adjacent pixels, $$x_{i,j} - x_{(i+1),j}$$. This cross-correlation operation serves as a discrete approximation of the first derivative in the horizontal direction. Mathematically, for an image function $$f(i,j)$$, its derivative is $$-\partial_i f(i,j) = \lim_{\epsilon \to 0} \frac{f(i,j) - f(i+\epsilon,j)}{\epsilon}$$. By applying this kernel, the output is zero where adjacent pixels are identical and non-zero at boundaries, effectively detecting edges.

Convolution Kernel as a Finite Difference Operator

As an example of padding in a convolutional layer, consider a $$3 	imes 3$$ input tensor padded with zero values to increase its size to $$5 	imes 5$$. If we apply a two-dimensional cross-correlation using a $$2 	imes 2$$ kernel on this padded input, the resulting output will be a $$4 	imes 4$$ matrix. The added zero pixels do not change the sum of the products when the convolution window covers them.

Example of Two-Dimensional Cross-Correlation with Padding

In convolutional operations, there are standard baseline configurations that are assumed unless specified otherwise. By default, the padding is set to $$0$$ (meaning no extra border pixels are added) and the stride is set to $$1$$ (meaning the convolution window slides exactly one element at a time).

Default Padding and Stride Values

When the input data contains multiple channels, denoted as $$c_	extrm{i}$$, the convolution kernel must have the same number of input channels to perform cross-correlation. If the kernel's two-dimensional spatial window shape is $$k_	extrm{h} 	imes k_	extrm{w}$$, a kernel tensor of shape $$k_	extrm{h} 	imes k_	extrm{w}$$ is required for every input channel. Concatenating these $$c_	extrm{i}$$ tensors yields a convolution kernel with an overall shape of $$c_	extrm{i} 	imes k_	extrm{h} 	imes k_	extrm{w}$$.

Multi-Channel Convolution Kernel Structure

In contrast to a regular convolution that reduces input elements via a kernel, the transposed convolution broadcasts input elements via the kernel, producing an output that is generally larger than the input. For a basic transposed convolution with a stride of $$1$$ and no padding, an $$n_h 	imes n_w$$ input tensor and a $$k_h 	imes k_w$$ kernel are used to produce an $$(n_h + k_h - 1) 	imes (n_w + k_w - 1)$$ output tensor. For each element in the input tensor, it is multiplied by the entire kernel to produce a $$k_h 	imes k_w$$ tensor. This resulting tensor replaces a corresponding portion of an intermediate tensor initialized with zeros, based on the element's original position. The final output is formed by summing all these intermediate results. For example, if both the input tensor and the kernel are $$2 	imes 2$$ matrices with values $$[[0, 1], [2, 3]]$$, the basic transposed convolution yields a $$3 	imes 3$$ output tensor with values $$[[0, 0, 1], [0, 4, 6], [4, 12, 9]]$$.

Learn Before

Related

Learn After