Modern reproductions of LeNet-5 replace the original Gaussian activation layer (also known as the Gaussian decoder or Euclidean Radial Basis Function output) with a softmax layer. This substitution greatly simplifies the implementation because the Gaussian decoder is rarely used in contemporary practice. Apart from this change, the reproduced network matches the original LeNet-5 architecture.

Claude

LeNet-5 architecture

Dive into Deep Learning

The convolutional encoder in LeNet-5 consists of two repeated units, each containing three operations: a convolutional layer, a sigmoid activation function, and an average pooling layer. Each convolutional layer uses a $$5 	imes 5$$ kernel. The first convolutional layer produces $$6$$ output channels, while the second produces $$16$$. After each convolution and activation, a $$2 	imes 2$$ average pooling operation with stride $$2$$ halves both the height and width of the representation, reducing the spatial dimensionality by a factor of $$4$$ per pooling step. The convolutional block maps spatially arranged inputs to a progressively increasing number of two-dimensional feature maps while decreasing spatial resolution. Its output has shape (batch size, number of channels, height, width).

Note that while ReLU and max-pooling perform better in modern networks, they had not yet been discovered at the time LeNet was designed.

LeNet-5 Convolutional Block

The dense (fully connected) block of LeNet-5 receives the output of the convolutional encoder. To bridge the two parts, the four-dimensional tensor output (batch size, channels, height, width) must be flattened into a two-dimensional representation: the first dimension indexes examples in the minibatch, and the second provides a flat vector for each example. LeNet-5's dense block consists of three fully connected layers with $$120$$, $$84$$, and $$10$$ output units, respectively. The final $$10$$-dimensional output layer corresponds to the number of possible output classes in the classification task.

LeNet-5 Dense Block

Passing a single-channel $$28 	imes 28$$ image through LeNet-5 produces the following shapes at each layer:

1. Conv2d ($$5 	imes 5$$, padding $$2$$, $$6$$ channels): output $$1 	imes 6 	imes 28 	imes 28$$
2. Sigmoid: output $$1 	imes 6 	imes 28 	imes 28$$
3. AvgPool2d ($$2 	imes 2$$, stride $$2$$): output $$1 	imes 6 	imes 14 	imes 14$$
4. Conv2d ($$5 	imes 5$$, no padding, $$16$$ channels): output $$1 	imes 16 	imes 10 	imes 10$$
5. Sigmoid: output $$1 	imes 16 	imes 10 	imes 10$$
6. AvgPool2d ($$2 	imes 2$$, stride $$2$$): output $$1 	imes 16 	imes 5 	imes 5$$
7. Flatten: output $$1 	imes 400$$
8. Linear ($$120$$) + Sigmoid: output $$1 	imes 120$$
9. Linear ($$84$$) + Sigmoid: output $$1 	imes 84$$
10. Linear ($$10$$): output $$1 	imes 10$$

The first convolutional layer uses $$2$$ pixels of padding to preserve the spatial dimensions of the $$28 	imes 28$$ input, compensating for the reduction that a $$5 	imes 5$$ kernel would otherwise cause. The second convolutional layer uses no padding, reducing height and width by $$4$$ pixels each. As the stack deepens, the number of channels increases ($$1 	o 6 	o 16$$) while spatial dimensions shrink, until flattening yields a $$400$$-element vector.

LeNet-5 Layer-by-Layer Shape Trace

Softmax Replacement of Gaussian Activation in LeNet-5

A key factor distinguishing early convolutional networks like LeNet-5 from later, substantially deeper architectures such as ResNet is the availability of computational resources. As hardware capabilities grew—particularly through advances in GPU processing—researchers were able to design and train networks with many more layers, skip connections, and sophisticated modules. Thus, while LeNet-5 and ResNet both belong to the CNN family and share core design principles, the primary driver behind the leap in architectural complexity was the dramatic increase in accessible computation, not a fundamental change in the underlying learning paradigm.

Learn Before

Related