Formula

Gradient Magnitude and Eigendecomposition

By analyzing a quadratic optimization problem in its eigendecomposed form fˉ(xˉ)\bar{f}(\bar{\mathbf{x}}), the gradient can be expressed as xˉfˉ(xˉ)=Λxˉ+cˉ=Λ(xˉxˉ0)\partial_{\bar{\mathbf{x}}} \bar{f}(\bar{\mathbf{x}}) = \boldsymbol{\Lambda} \bar{\mathbf{x}} + \bar{\mathbf{c}} = \boldsymbol{\Lambda} \left(\bar{\mathbf{x}} - \bar{\mathbf{x}}_0\right), where xˉ0\bar{\mathbf{x}}_0 is the true minimizer. This relationship demonstrates that the magnitude of the gradient is directly proportional to both the eigenvalue matrix Λ\boldsymbol{\Lambda} and the current distance from the optimal solution. When the distance from optimality remains relatively stable, the gradient's magnitude alone provides sufficient proportional information regarding the underlying scale of the Hessian matrix.

0

1

Updated 2026-05-15

Contributors are:

Who are from:

Tags

D2L

Dive into Deep Learning @ D2L