The fundamental intuition behind a piecewise constant learning rate schedule is to allow the optimization process to proceed with a steady learning rate until it reaches a stationary point or plateau regarding the distribution of weight vectors. By abruptly dropping the learning rate only after efficiently converging to this suitable solution, the model reduces the inherent variance of its parameters. This approach enables the discovery of a higher quality proxy to a good local minimum and slightly better overall solutions.

Claude

A piecewise constant learning rate schedule, often referred to as a multi-factor scheduler, maintains a steady learning rate for predefined intervals of training and abruptly drops the rate at specific milestones. Mathematically, given a set of milestone times $$s$$ (such as $$s = \{5, 10, 20\}$$), the learning rate is updated according to the rule $$\eta_{t+1} \leftarrow \eta_t \cdot \alpha$$ whenever the current step $$t \in s$$, where $$\alpha$$ is the designated decay factor.

Piecewise Constant Learning Rate Schedule

Dive into Deep Learning

Intuition Behind Piecewise Constant Learning Rate Schedules

A multi-factor (or piecewise constant) learning rate scheduler decreases the learning rate by a specific factor at predefined milestone epochs. While modern frameworks offer built-in solutions—such as `torch.optim.lr_scheduler.MultiStepLR` in PyTorch and `lr_scheduler.MultiFactorScheduler` in MXNet—this strategy can also be implemented from scratch. The following Python code demonstrates a custom scheduler that multiplies the base learning rate by a given factor whenever the current epoch is found in the predefined set of milestone steps:

```python
class MultiFactorScheduler:
    def __init__(self, step, factor, base_lr):
        self.step = step
        self.factor = factor
        self.base_lr = base_lr

    def __call__(self, epoch):
        if epoch in self.step:
            self.base_lr = self.base_lr * self.factor
            return self.base_lr
        else:
            return self.base_lr
```

Multi-Factor Learning Rate Scheduler Implementation

To stabilize optimization and achieve better convergence during fine-tuning, a step learning rate scheduler is often employed to periodically reduce the learning rate. By defining a period (lr_period) and a decay factor (lr_decay), tools like torch.optim.lr_scheduler.StepLR automatically multiply the learning rate by the decay factor after every specified number of epochs, ensuring smaller optimization steps as the model approaches a local minimum on the target dataset.

Step Learning Rate Scheduler in Fine-Tuning

To train the image classification model and select optimal hyperparameters based on validation set performance, a comprehensive training function is defined. This function orchestrates the training loop over a specified number of epochs using data parallelism across multiple devices. It initializes an optimization algorithm, such as stochastic gradient descent (SGD) with momentum and weight decay, and incorporates a step learning rate scheduler to periodically decay the learning rate by a specified factor. Within each epoch, the function iterates through training mini-batches to update parameters and accumulate training loss and accuracy. If a validation iterator is provided, it evaluates the model's accuracy on the hold-out validation set, plotting these metrics dynamically to monitor the model's generalization progress.

Learn Before

Related