To empirically demonstrate the impact of a cosine learning rate schedule, a neural network can be trained using an optimizer configured with this scheduler. The schedule dictates that the learning rate decreases gently at the beginning, drops more steeply, and then refines the solution at the end with a very small step size. While this approach can lead to improved training results and final accuracy—particularly in computer vision tasks—such improvements are heuristic and not strictly guaranteed across all models and datasets.

```python
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.3)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
      scheduler)
```

Claude

A cosine learning rate schedule, proposed by Loshchilov and Hutter (2016), dynamically adjusts the learning rate by following the shape of a cosine curve. It relies on the observation that the learning rate should not decrease too drastically at the beginning of training, and that the solution should be refined at the end using a very small learning rate. For learning rates in the range $$t \in [0, T]$$, this results in a schedule with the functional form:

$$\eta_t = \eta_T + \frac{\eta_0 - \eta_T}{2} \left(1 + \cos(\pi t/T)\right)$$

Here, $$\eta_0$$ is the initial learning rate and $$\eta_T$$ is the target rate at the maximum update step $$T$$. For steps $$t > T$$, the learning rate is simply pinned to $$\eta_T$$ without increasing it again.

Cosine Learning Rate Schedule

Dive into Deep Learning

A cosine learning rate scheduler can be implemented from scratch as a custom Python class. The class calculates the decayed learning rate based on the current step according to a cosine curve. It optionally includes a warmup phase where the learning rate increases linearly. Once past the warmup steps but within the maximum update steps, the learning rate is determined by the formula $$\eta_t = \eta_T + \frac{\eta_0 - \eta_T}{2} \left(1 + \cos\left(\frac{\pi (t - t_{\text{warmup}})}{T_{\text{max\_steps}}}\right)\right)$$. The following code demonstrates this implementation and plots the resulting schedule:

```python
class CosineScheduler:
    def __init__(self, max_update, base_lr=0.01, final_lr=0,
               warmup_steps=0, warmup_begin_lr=0):
        self.base_lr_orig = base_lr
        self.max_update = max_update
        self.final_lr = final_lr
        self.warmup_steps = warmup_steps
        self.warmup_begin_lr = warmup_begin_lr
        self.max_steps = self.max_update - self.warmup_steps

    def get_warmup_lr(self, epoch):
        increase = (self.base_lr_orig - self.warmup_begin_lr) \
                       * float(epoch) / float(self.warmup_steps)
        return self.warmup_begin_lr + increase

    def __call__(self, epoch):
        if epoch < self.warmup_steps:
            return self.get_warmup_lr(epoch)
        if epoch <= self.max_update:
            self.base_lr = self.final_lr + (
                self.base_lr_orig - self.final_lr) * (1 + math.cos(
                math.pi * (epoch - self.warmup_steps) / self.max_steps)) / 2
        return self.base_lr

scheduler = CosineScheduler(max_update=20, base_lr=0.3, final_lr=0.01)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
```

Learn Before

Related