Sometimes, higher order derivatives are needed for our models to learn. If we needed the second order derivatives, we could use the Hessian matrix. However, there are often millions or even billions of parameters in our models, so the Hessian is extremely difficult to calculate and represent.


For some function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ with a Hessian $\bold{H}$, and an arbitrary vector $v$:

$\bold{H}v=\nabla_{\bold{x}}[(\nabla_{\bold{x}}f(x))^{\top}v]$

Alternative to the Hessian (Krylov Methods)

First-order optimization algorithms rely solely on the value and gradient of the objective function. In contrast, second-order optimization algorithms also utilize information about the function's curvature, often represented by the Hessian matrix. By accounting for curvature, these methods can automatically adjust the optimization step, providing a way to circumvent the difficulties of manually tuning a learning rate.

Second-Order Optimization Algorithm

The second derivative of a function in a specific direction, represented by a unit vector $$d$$, is given by $$d^T H d$$, where $$H$$ is the Hessian matrix of the function.

Second Derivative in a Specific Direction

Hessian matrix is defined as a matrix containing second derivatives of function having multiple input dimensions. It's defined such that
$H(f)(x)_{i,j} = \frac{\partial^2}{\partial x_i \partial x_j}f(x)$

University of Connecticut

University of Michigan - Ann Arbor

Claude

The derivative of a function $$f: \mathbb{R} ightarrow \mathbb{R}$$ at a point $$x$$ represents the rate of change in the function with respect to changes in its arguments. Formally, it is defined as the limit of the ratio between a perturbation $$h$$ and the change in the function value as $$h$$ approaches zero: $$f'(x) = \lim_{h ightarrow 0} \frac{f(x+h) - f(x)}{h}$$. The derivative essentially measures how rapidly a function's value would increase or decrease given an infinitesimally small adjustment to its parameter.

Derivative of a Scalar Function

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

On a straight line, the function's derivative...

Gradient descent is a fundamental optimization algorithm that leverages gradients to minimize a model's loss function. Because the gradient of a function points in the direction of steepest ascent, moving the model's parameters in the opposite direction iteratively lowers the loss. Each step of such gradient-based optimization algorithms requires calculating the exact gradient of the loss with respect to the parameters.

Gradient Descent

https://www.wired.com/2015/04/crash-course-derivatives/

A crash course of derivatives

A derivative of a derivative is known as second derivative. Second derivative is often used to measure curvature.

Second Derivative

Hessian Matrix 

A Lipschitz continuous function is a functionfwhose rateof change is bounded by a Lipschitz constant $L$:
$L = \forall x \forall y. |f(x)-f(y)| \leq L ||x-y||_2$

Lipschitz Continuous

Functions that are composed of other differentiable functions can be differentiated using systematic rules. For differentiable functions $$f(x)$$ and $$g(x)$$ and a constant $$C$$, the primary rules are:
- Constant multiple rule: $$\frac{d}{dx} [C f(x)] = C \frac{d}{dx} f(x)$$
- Sum rule: $$\frac{d}{dx} [f(x) + g(x)] = \frac{d}{dx} f(x) + \frac{d}{dx} g(x)$$
- Product rule: $$\frac{d}{dx} [f(x) g(x)] = f(x) \frac{d}{dx} g(x) + g(x) \frac{d}{dx} f(x)$$
- Quotient rule: $$\frac{d}{dx} \frac{f(x)}{g(x)} = \frac{g(x) \frac{d}{dx} f(x) - f(x) \frac{d}{dx} g(x)}{g^2(x)}$$

Differentiation Rules

Several fundamental functions have well-known derivatives that serve as building blocks for more complex calculus operations. For any constant $$C$$ and any power $$n 
eq 0$$, the standard derivatives are:
- Constant: $$\frac{d}{dx} C = 0$$
- Power: $$\frac{d}{dx} x^n = n x^{n-1}$$
- Exponential: $$\frac{d}{dx} e^x = e^x$$
- Natural Logarithm: $$\frac{d}{dx} \ln x = x^{-1}$$

Derivatives of Common Functions

For functions of a single variable, the chain rule is used to compute the derivative of deeply nested functions. Suppose that $$y = f(g(x))$$ is a composite function, and that the underlying functions $$y = f(u)$$ and $$u = g(x)$$ are both differentiable. The chain rule states that the derivative of $$y$$ with respect to $$x$$ is the product of the derivative of $$y$$ with respect to $$u$$ and the derivative of $$u$$ with respect to $$x$$:

$$\frac{dy}{dx} = \frac{dy}{du} \frac{du}{dx}$$

Chain Rule for Single-Variable Functions

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Jacobian Matrix

The partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^	extrm{th}$$ parameter $$x_i$$ measures how the function changes as $$x_i$$ increases while treating all other variables as constants. Formally, it is defined as the limit: $$\frac{\partial y}{\partial x_i} = \lim_{h ightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$.

Partial Derivative

The gradient of a scalar-valued function with respect to a vector $$\mathbf{x}$$ is itself a vector-valued output that has the identical shape as the input vector $$\mathbf{x}$$. This property ensures that each element of the gradient vector directly corresponds to the partial derivative of the scalar output with respect to the matching element in the input vector.

Gradient of a Scalar-Valued Function with Respect to a Vector

Denote the function as $f(x)$, $g$ is the gradient and $H$ is is the Hessian at $x^{(0)}$. We calculate the new point $x = x^{(0)} - \epsilon g$. We can obtian that $f(x^{(0)} - \epsilon g) \approx f(x^{(0)}) - \epsilon g^Tg + \frac{1}{2} \epsilon^2 g^THg$ According to the above equation, the optimal step size when $g^THg$ is positive is $\epsilon^* = \frac{g^Tg}{g^THg}$

Learn Before

Related

Learn After