Optimizer and Learning Rate Schedule
The training process utilized the Adam optimizer with hyperparameter settings , , and .
The learning rate was modulated dynamically across steps according to the following formula:
This schedule increases the learning rate linearly during the initial phase. Following the warm-up period, the learning rate decays proportionally to the inverse square root of the step number (). In the reported experiments, .
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Transformer Training, Regularization, and Optimization - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
Match each optimizer hyperparameter or schedule parameter to its specific value used in the reported Transformer experiments.
During the initial phase defined by warmup_steps, the Transformer schedule modulates the learning rate such that it increases ___ with respect to the step number.
Order the trajectory of the learning rate across training steps from earliest to latest.
Explain why Model B has a lower learning rate than Model A throughout training according to the scheduling formula, and calculate the exact ratio of Model B's learning rate to Model A's learning rate (lrate_B / lrate_A).