To empirically study learning rate scheduling, a computationally efficient yet nontrivial toy problem can be constructed. A common setup involves training a modernized LeNet architecture on the Fashion-MNIST dataset. This modernized LeNet updates the classic design by replacing sigmoid activations with ReLU activations and substituting AveragePooling operations with MaxPooling.

Learning Rate Scheduler Toy Problem

The classic LeNet-5 neural network can be modernized to improve performance and reflect contemporary deep learning practices. This modernization primarily involves two architectural updates: replacing the original sigmoid activation functions with Rectified Linear Unit (ReLU) activations, and substituting average pooling operations with max pooling. When applied to classification tasks like Fashion-MNIST, this slightly modernized LeNet provides an efficient, nontrivial baseline model for evaluating training dynamics and optimization algorithms.

Claude

LeNet-5 is a pioneering convolutional neural network consisting of two main parts: (i) a convolutional encoder with two convolutional layers, and (ii) a dense block with three fully connected layers. The input is a handwritten digit image, and the output is a probability distribution over $$10$$ possible classes.

Convolutional encoder: Each convolutional block applies a $$5 	imes 5$$ kernel followed by a sigmoid activation function and a $$2 	imes 2$$ average pooling operation (stride $$2$$). The first convolutional layer produces $$6$$ output channels, and the second produces $$16$$ output channels. Each pooling operation reduces spatial dimensionality by a factor of $$4$$ (halving both height and width). The convolutional block's output shape is (batch size, number of channels, height, width).

Dense block: The four-dimensional output of the convolutional encoder is flattened into a two-dimensional representation where the first dimension indexes examples in the minibatch and the second gives each example's flat vector. Three fully connected layers with $$120$$, $$84$$, and $$10$$ outputs respectively form the dense block, with the $$10$$-dimensional final layer corresponding to the number of output classes.

Weights are initialized using Xavier initialization.

LeNet-5 remains architecturally meaningful today. When evaluated on Fashion-MNIST, LeNet-5 error rates are much closer to those of advanced architectures such as ResNet (Section 8.6) than to those achievable with basic MLPs (Section 5.2). This demonstrates the substantial leap in performance that convolutional architectures introduced over fully connected networks.

LeNet-5 Architecture

Dive into Deep Learning

- Most commonly used to recognize handwritten and machine printed characters, e.g., recognizing the handwritten numbers on bank checks.

LeNet-5 Convolutional Neural Network Applications

The convolutional encoder in LeNet-5 consists of two repeated units, each containing three operations: a convolutional layer, a sigmoid activation function, and an average pooling layer. Each convolutional layer uses a $$5 	imes 5$$ kernel. The first convolutional layer produces $$6$$ output channels, while the second produces $$16$$. After each convolution and activation, a $$2 	imes 2$$ average pooling operation with stride $$2$$ halves both the height and width of the representation, reducing the spatial dimensionality by a factor of $$4$$ per pooling step. The convolutional block maps spatially arranged inputs to a progressively increasing number of two-dimensional feature maps while decreasing spatial resolution. Its output has shape (batch size, number of channels, height, width).

Note that while ReLU and max-pooling perform better in modern networks, they had not yet been discovered at the time LeNet was designed.

LeNet-5 Convolutional Block

The dense (fully connected) block of LeNet-5 receives the output of the convolutional encoder. To bridge the two parts, the four-dimensional tensor output (batch size, channels, height, width) must be flattened into a two-dimensional representation: the first dimension indexes examples in the minibatch, and the second provides a flat vector for each example. LeNet-5's dense block consists of three fully connected layers with $$120$$, $$84$$, and $$10$$ output units, respectively. The final $$10$$-dimensional output layer corresponds to the number of possible output classes in the classification task.

LeNet-5 Dense Block

Passing a single-channel $$28 	imes 28$$ image through LeNet-5 produces the following shapes at each layer:

1. Conv2d ($$5 	imes 5$$, padding $$2$$, $$6$$ channels): output $$1 	imes 6 	imes 28 	imes 28$$
2. Sigmoid: output $$1 	imes 6 	imes 28 	imes 28$$
3. AvgPool2d ($$2 	imes 2$$, stride $$2$$): output $$1 	imes 6 	imes 14 	imes 14$$
4. Conv2d ($$5 	imes 5$$, no padding, $$16$$ channels): output $$1 	imes 16 	imes 10 	imes 10$$
5. Sigmoid: output $$1 	imes 16 	imes 10 	imes 10$$
6. AvgPool2d ($$2 	imes 2$$, stride $$2$$): output $$1 	imes 16 	imes 5 	imes 5$$
7. Flatten: output $$1 	imes 400$$
8. Linear ($$120$$) + Sigmoid: output $$1 	imes 120$$
9. Linear ($$84$$) + Sigmoid: output $$1 	imes 84$$
10. Linear ($$10$$): output $$1 	imes 10$$

The first convolutional layer uses $$2$$ pixels of padding to preserve the spatial dimensions of the $$28 	imes 28$$ input, compensating for the reduction that a $$5 	imes 5$$ kernel would otherwise cause. The second convolutional layer uses no padding, reducing height and width by $$4$$ pixels each. As the stack deepens, the number of channels increases ($$1 	o 6 	o 16$$) while spatial dimensions shrink, until flattening yields a $$400$$-element vector.

Learn Before

Related

Learn After