A computational graph is a directed dependency graph where nodes represent mathematical operations or variables, mapping the functional dependency from inputs to outputs. When evaluating a vector-valued function, this graph is traversed in a forward direction. Conversely, to compute gradients automatically, the graph is traced in a backwards direction from output to input, which involves multiplying matrices of partial derivatives along the paths.

Computational Graph

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Jacobian Matrix

The derivative of a vector $$\mathbf{u}$$ (consisting of $$m$$ variables) with respect to a vector $$\mathbf{x}$$ (consisting of $$n$$ variables) is an $$n 	imes m$$ matrix, commonly denoted as $$\mathbf{A}$$. This matrix compiles all the individual partial derivatives of the components of $$\mathbf{u}$$ with respect to the components of $$\mathbf{x}$$.

Claude

The partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^	extrm{th}$$ parameter $$x_i$$ measures how the function changes as $$x_i$$ increases while treating all other variables as constants. Formally, it is defined as the limit: $$\frac{\partial y}{\partial x_i} = \lim_{h ightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$.

Partial Derivative

Dive into Deep Learning

There are several equivalent mathematical notations used to represent the partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^{	extrm{th}}$$ parameter, $$x_i$$. Common conventions include fractional forms like $$\frac{\partial y}{\partial x_i}$$ and $$\frac{\partial f}{\partial x_i}$$, subscripted differential operators such as $$\partial_{x_i} f$$, $$\partial_i f$$, $$D_i f$$, and $$D_{x_i} f$$, as well as subscripted function names like $$f_{x_i}$$ and $$f_i$$.

Learn Before

Related

Learn After