When configuring the optimizer for a fine-tuning task where the base network is frozen, it is necessary to explicitly instruct the optimizer to only update the unfrozen weights. In frameworks like PyTorch, this is achieved by filtering the model's parameters and passing only those that require gradients—using a generator expression like (param for param in net.parameters() if param.requires_grad)—to the optimizer (e.g., Stochastic Gradient Descent).

Claude

When utilizing a pretrained model solely as a feature extractor, its parameters are explicitly frozen to prevent them from being updated during training. Because these feature extraction layers remain fixed, the optimization algorithm does not need to compute or store their gradients during backpropagation. This approach substantially reduces the overall training time and minimizes the memory footprint required for storing gradients, making it highly efficient for adapting large models to new tasks.

Computational Benefits of Freezing Pretrained Parameters

Dive into Deep Learning

To adapt a pretrained model for a new classification task, its original output layer is bypassed, and the intermediate representations—or extracted features—are fed into a newly defined custom output network. Instead of a simple linear mapping, this custom network can be a small-scale architecture, such as a stack of multiple fully connected layers with non-linear activations. During the forward pass, the extracted features serve as the direct input to this custom output network, which is trained from scratch to generate the final predictions.

Learn Before

Related