While the Jacobian matrix provides the full derivative of a vector output, it is more common in machine learning to sum the gradients of each component of an output vector $$\mathbf{y}$$ with respect to the full input vector $$\mathbf{x}$$. This reduction yields a gradient vector that has the exact same shape as $$\mathbf{x}$$, a technique frequently used to aggregate gradients calculated individually for each training example in a batch.

Gradient Reduction for Non-Scalar Outputs

When differentiating higher-order tensors, the dimensionality of the resulting derivative expands. Just as the derivative of a vector with respect to another vector yields a matrix (the Jacobian), calculating the derivative of a higher-order tensor $$\mathbf{y}$$ with respect to a higher-order tensor $$\mathbf{x}$$ produces an even higher-order tensor. This resulting tensor contains the partial derivatives of each component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Derivative of Higher-Order Tensors

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

University of Michigan - Ann Arbor

Claude

The derivative of a function $$f: \mathbb{R} ightarrow \mathbb{R}$$ at a point $$x$$ represents the rate of change in the function with respect to changes in its arguments. Formally, it is defined as the limit of the ratio between a perturbation $$h$$ and the change in the function value as $$h$$ approaches zero: $$f'(x) = \lim_{h ightarrow 0} \frac{f(x+h) - f(x)}{h}$$. The derivative essentially measures how rapidly a function's value would increase or decrease given an infinitesimally small adjustment to its parameter.

Derivative of a Scalar Function

The derivative of a vector $$\mathbf{u}$$ (consisting of $$m$$ variables) with respect to a vector $$\mathbf{x}$$ (consisting of $$n$$ variables) is an $$n 	imes m$$ matrix, commonly denoted as $$\mathbf{A}$$. This matrix compiles all the individual partial derivatives of the components of $$\mathbf{u}$$ with respect to the components of $$\mathbf{x}$$.

Derivative of a Vector with Respect to a Vector

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

On a straight line, the function's derivative...

Gradient descent is a fundamental optimization algorithm that leverages gradients to minimize a model's loss function. Because the gradient of a function points in the direction of steepest ascent, moving the model's parameters in the opposite direction iteratively lowers the loss. Each step of such gradient-based optimization algorithms requires calculating the exact gradient of the loss with respect to the parameters.

Gradient Descent

https://www.wired.com/2015/04/crash-course-derivatives/

A crash course of derivatives

Learning rate is a positive scalar determining the size of the step in method of steepest descent. 

Learning Rate

A derivative of a derivative is known as second derivative. Second derivative is often used to measure curvature.

Second Derivative

Hessian matrix is defined as a matrix containing second derivatives of function having multiple input dimensions. It's defined such that
$H(f)(x)_{i,j} = \frac{\partial^2}{\partial x_i \partial x_j}f(x)$

Hessian Matrix 

Denote the function as $f(x)$, $g$ is the gradient and $H$ is is the Hessian at $x^{(0)}$. We calculate the new point $x = x^{(0)} - \epsilon g$. We can obtian that
$f(x^{(0)} - \epsilon g) \approx f(x^{(0)}) - \epsilon g^Tg + \frac{1}{2} \epsilon^2 g^THg$ 
According to the above equation, the optimal step size when $g^THg$ is positive is
$\epsilon^* = \frac{g^Tg}{g^THg}$

Optimal Step Size according to Taylor Series Approximation

A Lipschitz continuous function is a functionfwhose rateof change is bounded by a Lipschitz constant $L$:
$L = \forall x \forall y. |f(x)-f(y)| \leq L ||x-y||_2$

Lipschitz Continuous

Functions that are composed of other differentiable functions can be differentiated using systematic rules. For differentiable functions $$f(x)$$ and $$g(x)$$ and a constant $$C$$, the primary rules are:
- Constant multiple rule: $$\frac{d}{dx} [C f(x)] = C \frac{d}{dx} f(x)$$
- Sum rule: $$\frac{d}{dx} [f(x) + g(x)] = \frac{d}{dx} f(x) + \frac{d}{dx} g(x)$$
- Product rule: $$\frac{d}{dx} [f(x) g(x)] = f(x) \frac{d}{dx} g(x) + g(x) \frac{d}{dx} f(x)$$
- Quotient rule: $$\frac{d}{dx} \frac{f(x)}{g(x)} = \frac{g(x) \frac{d}{dx} f(x) - f(x) \frac{d}{dx} g(x)}{g^2(x)}$$

Differentiation Rules

Several fundamental functions have well-known derivatives that serve as building blocks for more complex calculus operations. For any constant $$C$$ and any power $$n 
eq 0$$, the standard derivatives are:
- Constant: $$\frac{d}{dx} C = 0$$
- Power: $$\frac{d}{dx} x^n = n x^{n-1}$$
- Exponential: $$\frac{d}{dx} e^x = e^x$$
- Natural Logarithm: $$\frac{d}{dx} \ln x = x^{-1}$$

Derivatives of Common Functions

For functions of a single variable, the chain rule is used to compute the derivative of deeply nested functions. Suppose that $$y = f(g(x))$$ is a composite function, and that the underlying functions $$y = f(u)$$ and $$u = g(x)$$ are both differentiable. The chain rule states that the derivative of $$y$$ with respect to $$x$$ is the product of the derivative of $$y$$ with respect to $$u$$ and the derivative of $$u$$ with respect to $$x$$:

$$\frac{dy}{dx} = \frac{dy}{du} \frac{du}{dx}$$

Chain Rule for Single-Variable Functions

Jacobian Matrix

The partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^	extrm{th}$$ parameter $$x_i$$ measures how the function changes as $$x_i$$ increases while treating all other variables as constants. Formally, it is defined as the limit: $$\frac{\partial y}{\partial x_i} = \lim_{h ightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$.

Partial Derivative

The gradient of a scalar-valued function with respect to a vector $$\mathbf{x}$$ is itself a vector-valued output that has the identical shape as the input vector $$\mathbf{x}$$. This property ensures that each element of the gradient vector directly corresponds to the partial derivative of the scalar output with respect to the matching element in the input vector.

Gradient of a Scalar-Valued Function with Respect to a Vector

A computational graph is a directed dependency graph where nodes represent mathematical operations or variables, mapping the functional dependency from inputs to outputs. When evaluating a vector-valued function, this graph is traversed in a forward direction. Conversely, to compute gradients automatically, the graph is traced in a backwards direction from output to input, which involves multiplying matrices of partial derivatives along the paths.

Learn Before

Related

Learn After