Learn Before
During the initial phase defined by warmup_steps, the Transformer schedule modulates the learning rate such that it increases ___ with respect to the step number.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Transformer Training, Regularization, and Optimization - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Match each optimizer hyperparameter or schedule parameter to its specific value used in the reported Transformer experiments.
During the initial phase defined by warmup_steps, the Transformer schedule modulates the learning rate such that it increases ___ with respect to the step number.
Order the trajectory of the learning rate across training steps from earliest to latest.
Explain why Model B has a lower learning rate than Model A throughout training according to the scheduling formula, and calculate the exact ratio of Model B's learning rate to Model A's learning rate (lrate_B / lrate_A).