Momentum Convergence on a Scalar Quadratic
To analyze the convergence of the momentum method on a scalar quadratic function , the update equations for the position and velocity can be formulated as a coupled system: . The convergence behavior is entirely governed by the eigenvalues of this transition matrix. Analysis of this matrix reveals that the velocity converges when the hyperparameters satisfy . This feasible range is substantially larger than the constraint required for standard gradient descent, confirming that large momentum coefficients () safely permit much larger learning rates without divergence.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Related
Intuition behind Gradient Descent with Momentum
Identifying Gradient Descent Variants from Text Descriptions
Adam (Deep Learning Optimization Algorithm)
Origin of the Momentum Method
Velocity Initialization in Momentum Method
Momentum Convergence on a Scalar Quadratic
Gradient Descent with Momentum Pseudocode
Momentum Convergence on a Scalar Quadratic