Learn Before
Concept

Newton's Method

Newton's Method is a second-order optimization algorithm that relies on the second-order Taylor expansion of a multivariate function f(x)f(\mathbf{x}). By taking the expansion f(x+ϵ)=f(x)+ϵopablaf(x)+12ϵopHϵ+O(ϵ3)f(\mathbf{x} + \boldsymbol{\epsilon}) = f(\mathbf{x}) + \boldsymbol{\epsilon}^ op abla f(\mathbf{x}) + \frac{1}{2} \boldsymbol{\epsilon}^ op \mathbf{H} \boldsymbol{\epsilon} + \mathcal{O}(\|\boldsymbol{\epsilon}\|^3), where H=abla2f(x)\mathbf{H} = abla^2 f(\mathbf{x}) is the Hessian matrix, and setting the derivative with respect to the update step ϵ\boldsymbol{\epsilon} to zero (ablaf(x)+Hϵ=0 abla f(\mathbf{x}) + \mathbf{H} \boldsymbol{\epsilon} = 0), the algorithm derives the optimal update step as ϵ=H1ablaf(x)\boldsymbol{\epsilon} = -\mathbf{H}^{-1} abla f(\mathbf{x}). This approach requires computing and inverting the Hessian matrix to directly jump toward the function's minimum.

0

1

Updated 2026-05-15

Tags

Data Science

D2L

Dive into Deep Learning @ D2L