Learn Before
Factor Learning Rate Scheduler
A factor learning rate scheduler is an alternative to polynomial decay that utilizes a multiplicative reduction strategy. The learning rate is updated at each step using the equation , where . To prevent the learning rate from decaying beyond a reasonable lower bound, the update equation is typically modified to .
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Related
Effect of Learning Rate Scheduling on Overfitting
Polynomial Learning Rate Decay
Piecewise Constant Learning Rate Schedule
Cosine Learning Rate Schedule
Optimizer Warmup
Factor Learning Rate Scheduler
Explicit Learning Rate Adjustment Implementation
Learning Rate Scheduler Toy Problem
Square Root Learning Rate Scheduler
Transformer Adam Settings and Warmup–Inverse-Square-Root Learning Rate Formula