When a model has millions of parameters, the full Hessian matrix is computationally expensive to calculate and store. Krylov methods offer an alternative optimization approach by only requiring the product between the Hessian and an arbitrary vector. For a function $$f:\mathbb{R}^n\rightarrow \mathbb{R}$$ with a Hessian $$\mathbf{H}$$ and an arbitrary vector $$v$$, this Hessian-vector product can be evaluated using only gradient operations: $$\mathbf{H}v=\nabla_{\mathbf{x}}[(\nabla_{\mathbf{x}}f(x))^{\top}v]$$.

University of Connecticut

Google

Hessian matrix is defined as a matrix containing second derivatives of function having multiple input dimensions. It's defined such that
$H(f)(x)_{i,j} = \frac{\partial^2}{\partial x_i \partial x_j}f(x)$

Hessian Matrix 

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

First-order optimization algorithms rely solely on the value and gradient of the objective function. In contrast, second-order optimization algorithms also utilize information about the function's curvature, often represented by the Hessian matrix. By accounting for curvature, these methods can automatically adjust the optimization step, providing a way to circumvent the difficulties of manually tuning a learning rate.

Second-Order Optimization Algorithm

The second derivative of a function in a specific direction, represented by a unit vector $$d$$, is given by $$d^T H d$$, where $$H$$ is the Hessian matrix of the function.

Learn Before

Related