When training AlexNet, several key differences emerge compared to training the simpler LeNet-5 architecture. Because AlexNet is a much deeper and wider network, and because it processes higher-resolution images with more computationally costly convolutions, its training process is significantly slower. Consequently, training AlexNet typically requires the use of a smaller learning rate to ensure stable convergence, whereas LeNet-5 can often be trained with a larger learning rate and converges much faster due to its smaller size and lower-resolution inputs.

AlexNet and LeNet-5 Training Comparison

While the architectures of AlexNet and LeNet-5 share a similar foundational design mapping spatial inputs to fully connected layers, they exhibit several significant differences. First, AlexNet is much deeper, consisting of an $$8$$-layer structure ($$5$$ convolutional and $$3$$ fully connected layers) compared to the smaller LeNet-5. Second, AlexNet utilizes the Rectified Linear Unit (ReLU) activation function instead of LeNet's sigmoid function. Third, AlexNet controls the enormous capacity of its fully connected layers using dropout, whereas LeNet relies solely on weight decay. Finally, AlexNet employs extensive image augmentation during training to increase robustness, a technique not standard in the LeNet era.

Claude

Introduced in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, AlexNet was a pioneering $$8$$-layer Convolutional Neural Network (CNN) that won the 2012 ImageNet Large Scale Visual Recognition Challenge by a large margin. It demonstrated for the first time that features obtained through automatic learning could transcend manually-designed features, effectively breaking the previous paradigm in computer vision. Structurally, it is an evolutionary improvement over the earlier LeNet-5 architecture, sharing many architectural elements but scaled up significantly to leverage massive training datasets and faster Graphics Processing Units (GPUs). The network processes input images through a deep hierarchy of convolutional layers, max-pooling layers, and fully connected layers to generate predictions.

AlexNet Convolutional Neural Network

Dive into Deep Learning

- Most commonly used for image classification to sort images into different categories, e.g., keyword search on images.

AlexNet Convolutional Neural Network Applications

In the lowest layers of the AlexNet architecture, the model automatically learned feature extractors that strongly resembled traditional, hand-crafted image filters. These early-layer convolutional filters act as fundamental visual descriptors, such as edge, color, and texture detectors. By learning these foundational representations directly from the raw pixel data, the network can systematically build upon them in higher layers to represent increasingly complex structures and eventually recognize entire objects.

AlexNet Image Filters

AlexNet and LeNet-5 Architectural Comparison

To artificially increase the training sample size and reduce overfitting, the training loop of AlexNet employs extensive image augmentation techniques. These preprocessing steps include geometric and photometric transformations such as flipping, clipping, and color changes, which make the model significantly more robust to variations in input data.

AlexNet Image Augmentation

The architecture of AlexNet can be implemented in modern deep learning frameworks like PyTorch by sequentially composing convolutional, pooling, and fully connected layers. The network starts with a large $$11 \times 11$$ convolution with a stride of $$4$$, followed by ReLU activations and max-pooling operations. It then processes the feature maps through smaller $$5 \times 5$$ and $$3 \times 3$$ convolutions. Finally, the tensor is flattened and passed through two $$4096$$-dimensional fully connected layers, where dropout ($$p=0.5$$) is applied to prevent overfitting.

```python
class AlexNet(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(
            nn.LazyConv2d(96, kernel_size=11, stride=4, padding=1),
            nn.ReLU(), nn.MaxPool2d(kernel_size=3, stride=2),
            nn.LazyConv2d(256, kernel_size=5, padding=2), nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.LazyConv2d(384, kernel_size=3, padding=1), nn.ReLU(),
            nn.LazyConv2d(384, kernel_size=3, padding=1), nn.ReLU(),
            nn.LazyConv2d(256, kernel_size=3, padding=1), nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=2), nn.Flatten(),
            nn.LazyLinear(4096), nn.ReLU(), nn.Dropout(p=0.5),
            nn.LazyLinear(4096), nn.ReLU(), nn.Dropout(p=0.5),
            nn.LazyLinear(num_classes))
        self.net.apply(d2l.init_cnn)
```

AlexNet Implementation Code

Although AlexNet was originally trained on the large-scale ImageNet dataset, it is often trained on Fashion-MNIST for educational demonstrations because training an ImageNet model to convergence can take hours or days even on a modern GPU. However, Fashion-MNIST images have a lower resolution ($$28 	imes 28$$ pixels) compared to the ImageNet images ($$224 	imes 224$$ pixels) that AlexNet expects. To apply AlexNet directly to Fashion-MNIST, the images must be upsampled to $$224 	imes 224$$. While this upsampling allows the images to fit the network's architecture, it is generally not a smart practice in real-world applications because it simply increases the computational complexity without adding any new information.

Training AlexNet on Fashion-MNIST

The widespread academic adoption of deep neural networks following the introduction of AlexNet was initially delayed by a lack of efficient computational tools. In $$2012$$, implementing such deep architectures was a complex process requiring several months of manual labor. Early systems like Theano lacked many distinguishing features, while tools such as DistBelief and Caffe had not yet been created. It was the eventual release of more advanced systems, notably TensorFlow in $$2016$$, that dramatically changed the landscape by streamlining model development. Today, the same deep learning architectures that once took months to build can be concisely implemented in just a dozen lines of code using modern frameworks.

Historical Impact of Deep Learning Frameworks

A striking empirical observation when training the AlexNet architecture on datasets like Fashion-MNIST is the near absence of overfitting, despite a massive discrepancy between the model's capacity and the dataset size. For example, the final two fully connected layers of AlexNet contain over $$40$$ million learnable parameters, while the training dataset consists of only $$60,000$$ images. Yet, the training and validation losses remain virtually identical throughout the training process. This remarkable generalization is directly attributed to the improved regularization strategies inherent in the network's design, specifically the use of dropout.

Learn Before

Related

Learn After