The second derivative of a function in a specific direction, represented by a unit vector $$d$$, is given by $$d^T H d$$, where $$H$$ is the Hessian matrix of the function.

University of Michigan - Ann Arbor

Google

A derivative of a derivative is known as second derivative. Second derivative is often used to measure curvature.

Second Derivative

Hessian matrix is defined as a matrix containing second derivatives of function having multiple input dimensions. It's defined such that
$H(f)(x)_{i,j} = \frac{\partial^2}{\partial x_i \partial x_j}f(x)$

Hessian Matrix 

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

The second derivative can be used to determine whether a critical point is a local maximum, a local minimum, or a saddle point. We call such a test second derivative test.

Second Derivative Test

A function $f(x)$ is considered convex if the line segment connecting any two points on its graph lies on or above the graph itself. From a calculus perspective, this geometric property corresponds to a non-negative second derivative ($f''(x) \ge 0$), which signifies that the function's slope, given by the first derivative ($f'(x)$), is non-decreasing.

Convex Function

Second Derivative in a Specific Direction

Sometimes, higher order derivatives are needed for our models to learn. If we needed the second order derivatives, we could use the Hessian matrix. However, there are often millions or even billions of parameters in our models, so the Hessian is extremely difficult to calculate and represent.


For some function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ with a Hessian $\bold{H}$, and an arbitrary vector $v$:

$\bold{H}v=\nabla_{\bold{x}}[(\nabla_{\bold{x}}f(x))^{\top}v]$

Alternative to the Hessian (Krylov Methods)

First-order optimization algorithms rely solely on the value and gradient of the objective function. In contrast, second-order optimization algorithms also utilize information about the function's curvature, often represented by the Hessian matrix. By accounting for curvature, these methods can automatically adjust the optimization step, providing a way to circumvent the difficulties of manually tuning a learning rate.

Learn Before

Related