To train the image classification model and select optimal hyperparameters based on validation set performance, a comprehensive training function is defined. This function orchestrates the training loop over a specified number of epochs using data parallelism across multiple devices. It initializes an optimization algorithm, such as stochastic gradient descent (SGD) with momentum and weight decay, and incorporates a step learning rate scheduler to periodically decay the learning rate by a specified factor. Within each epoch, the function iterates through training mini-batches to update parameters and accumulate training loss and accuracy. If a validation iterator is provided, it evaluates the model's accuracy on the hold-out validation set, plotting these metrics dynamically to monitor the model's generalization progress.

CIFAR-10 Model Training Function Definition

To evaluate the predictions of the ResNet-18 model on the CIFAR-10 dataset, the cross-entropy loss function is utilized. In some framework implementations, this loss function is explicitly configured to compute the unreduced loss (e.g., `reduction="none"`), returning individual loss values for each example rather than a single aggregated scalar.

Claude

For a pair of a one-hot label vector $$\mathbf{y}$$ and a model's predicted probability distribution $$\hat{\mathbf{y}}$$ over $$q$$ classes, the cross-entropy loss function is defined as:

$$l(\mathbf{y}, \hat{\mathbf{y}}) = - \sum_{j=1}^q y_j \log \hat{y}_j$$

Because $$\mathbf{y}$$ is a one-hot vector, the sum vanishes for all but the coordinate corresponding to the true class. This loss is bounded below by $$0$$ (since probabilities cannot exceed $$1$$ and their negative logarithm cannot be lower than $$0$$), and it only equals $$0$$ if the model predicts the true label with absolute certainty. However, reaching a probability of exactly $$1$$ requires infinite logits, so the loss is never completely $$0$$ for finite weights. Conversely, assigning an output probability of $$0$$ to the true label would incur an infinite loss ($$-\log 0 = \infty$$).

Cross-Entropy Loss for Softmax Regression

Dive into Deep Learning

When calculating the cross-entropy loss for a batch of examples, it is computationally inefficient to use a for-loop to iterate over each example to evaluate the negative log-likelihood. Instead, advanced array indexing can be used to extract the model's predicted probability assigned to the true label for each example. Because the true labels $$\mathbf{y}$$ are typically provided as a vector of integer class indices, these indices can directly select the corresponding predicted probabilities from the prediction matrix $$\hat{\mathbf{y}}$$. This efficiently bypasses the need for explicitly multiplying a one-hot encoded label matrix by the predictions.

Learn Before

Related

Learn After