First-order optimization algorithms rely solely on the value and gradient of the objective function. In contrast, second-order optimization algorithms also utilize information about the function's curvature, often represented by the Hessian matrix. By accounting for curvature, these methods can automatically adjust the optimization step, providing a way to circumvent the difficulties of manually tuning a learning rate.

Second-Order Optimization Algorithm

Adaptive optimization methods aim to address the notoriously difficult problem of selecting an optimal learning rate, $$\eta$$. Because a learning rate that is too small leads to excessively slow progress and one that is too large causes oscillation or divergence, adaptive methods attempt to determine $$\eta$$ automatically or eliminate the need for manual tuning entirely during the optimization process.

Claude

When the learning rate $$\eta$$ is chosen to be too small, each gradient descent update moves the parameter $$x$$ only a tiny distance toward the optimum. This results in extremely slow progress, with the algorithm requiring a large number of iterations to reach a satisfactory solution. For instance, applying gradient descent to the quadratic $$f(x) = x^2$$ with $$\eta = 0.05$$ and starting from $$x = 10$$, the parameter value is still approximately $$3.49$$ after $$10$$ iterations—far from the optimal solution at $$x = 0$$. While a small learning rate ensures that the first-order Taylor approximation remains valid and the function value decreases at every step, the practical cost is an unacceptably slow convergence rate.

Effect of a Small Learning Rate on Gradient Descent

When the learning rate $$\eta$$ is set too high, the magnitude of the gradient step $$|\eta f'(x)|$$ can become large enough to invalidate the first-order Taylor expansion used to justify gradient descent. Specifically, the higher-order remainder term $$\mathcal{O}(\eta^2 (f'(x))^2)$$ is no longer negligible, so there is no guarantee that the function value will decrease after each update. In practice, the iterates overshoot the optimal solution and can diverge. For example, applying gradient descent to $$f(x) = x^2$$ with $$\eta = 1.1$$ starting from $$x = 5$$ causes $$x$$ to overshoot the minimum at $$x = 0$$ and gradually diverge, reaching approximately $$61.92$$ after $$10$$ iterations instead of converging.

Learn Before

Related

Learn After