When differentiating higher-order tensors, the dimensionality of the resulting derivative expands. Just as the derivative of a vector with respect to another vector yields a matrix (the Jacobian), calculating the derivative of a higher-order tensor $$\mathbf{y}$$ with respect to a higher-order tensor $$\mathbf{x}$$ produces an even higher-order tensor. This resulting tensor contains the partial derivatives of each component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Claude

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Jacobian Matrix

Dive into Deep Learning

While the Jacobian matrix provides the full derivative of a vector output, it is more common in machine learning to sum the gradients of each component of an output vector $$\mathbf{y}$$ with respect to the full input vector $$\mathbf{x}$$. This reduction yields a gradient vector that has the exact same shape as $$\mathbf{x}$$, a technique frequently used to aggregate gradients calculated individually for each training example in a batch.

Learn Before

Related