The convolutional encoder in LeNet-5 consists of two repeated units, each containing three operations: a convolutional layer, a sigmoid activation function, and an average pooling layer. Each convolutional layer uses a $$5 	imes 5$$ kernel. The first convolutional layer produces $$6$$ output channels, while the second produces $$16$$. After each convolution and activation, a $$2 	imes 2$$ average pooling operation with stride $$2$$ halves both the height and width of the representation, reducing the spatial dimensionality by a factor of $$4$$ per pooling step. The convolutional block maps spatially arranged inputs to a progressively increasing number of two-dimensional feature maps while decreasing spatial resolution. Its output has shape (batch size, number of channels, height, width).

Note that while ReLU and max-pooling perform better in modern networks, they had not yet been discovered at the time LeNet was designed.

LeNet-5 Convolutional Block

The dense (fully connected) block of LeNet-5 receives the output of the convolutional encoder. To bridge the two parts, the four-dimensional tensor output (batch size, channels, height, width) must be flattened into a two-dimensional representation: the first dimension indexes examples in the minibatch, and the second provides a flat vector for each example. LeNet-5's dense block consists of three fully connected layers with $$120$$, $$84$$, and $$10$$ output units, respectively. The final $$10$$-dimensional output layer corresponds to the number of possible output classes in the classification task.

LeNet-5 Dense Block

Passing a single-channel $$ 28 \times 28 $$ image through LeNet-5 produces the following shapes at each layer:

1. Conv2d ($$ 5 \times 5 $$, padding $$ 2 $$, $$ 6 $$ channels): output $$ 1 \times 6 \times 28 \times 28 $$
2. Sigmoid: output $$ 1 \times 6 \times 28 \times 28 $$
3. AvgPool2d ($$ 2 \times 2 $$, stride $$ 2 $$): output $$ 1 \times 6 \times 14 \times 14 $$
4. Conv2d ($$ 5 \times 5 $$, no padding, $$ 16 $$ channels): output $$ 1 \times 16 \times 10 \times 10 $$
5. Sigmoid: output $$ 1 \times 16 \times 10 \times 10 $$
6. AvgPool2d ($$ 2 \times 2 $$, stride $$ 2 $$): output $$ 1 \times 16 \times 5 \times 5 $$
7. Flatten: output $$ 1 \times 400 $$
8. Linear ($$ 120 $$) + Sigmoid: output $$ 1 \times 120 $$
9. Linear ($$ 84 $$) + Sigmoid: output $$ 1 \times 84 $$
10. Linear ($$ 10 $$): output $$ 1 \times 10 $$

The first convolutional layer uses $$ 2 $$ pixels of padding to preserve the spatial dimensions of the $$ 28 \times 28 $$ input, compensating for the reduction that a $$ 5 \times 5 $$ kernel would otherwise cause. The second convolutional layer uses no padding, reducing height and width by $$ 4 $$ pixels each. As the stack deepens, the number of channels increases ($$ 1 \to 6 \to 16 $$) while spatial dimensions shrink, until flattening yields a $$ 400 $$-element vector.

LeNet-5 Layer-by-Layer Shape Trace

Modern reproductions of LeNet-5 replace the original Gaussian activation layer (also known as the Gaussian decoder or Euclidean Radial Basis Function output) with a softmax layer. This substitution greatly simplifies the implementation because the Gaussian decoder is rarely used in contemporary practice. Apart from this change, the reproduced network matches the original LeNet-5 architecture.

Softmax Replacement of Gaussian Activation in LeNet-5

A key factor distinguishing early convolutional networks like LeNet-5 from later, substantially deeper architectures such as ResNet is the availability of computational resources. As hardware capabilities grew—particularly through advances in GPU processing—researchers were able to design and train networks with many more layers, skip connections, and sophisticated modules. Thus, while LeNet-5 and ResNet both belong to the CNN family and share core design principles, the primary driver behind the leap in architectural complexity was the dramatic increase in accessible computation, not a fundamental change in the underlying learning paradigm.

Computational Scale Enabling Architectural Complexity

University of Michigan - Ann Arbor

Three classic networks

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Learn Before

Related

Learn After

Learn After

Related