To unfold a computational graph is to unfold a recursive or recurrent computation into repetitive structure, typically corresponding to a chain of events.

Unfolding Operation

During automatic differentiation, it is sometimes necessary to exclude specific calculations from the recorded computational graph. This is often required when inputs generate auxiliary intermediate variables that should not contribute to the gradient computation. To achieve this, the computational graph associated with these intermediate terms is detached from the final result. A new variable is instantiated with the same numerical value, but its provenance—the record of operations that created it—is completely erased. Consequently, this new variable acts as a constant with no ancestors in the graph, preventing gradients from flowing back through it to earlier variables.

Detaching Computation

While automatic differentiation is used to optimize deep learning models in a statistical sense (e.g., through gradient descent), the computational optimization of the autograd libraries themselves is a vital area of framework design. Framework developers leverage techniques from compilers and graph manipulation to execute gradient computations in the most expedient and memory-efficient manner possible.

Optimization of Automatic Differentiation Libraries

Backpropagation is a systematic computational procedure for applying the chain rule to calculate gradients automatically. It operates by traversing a computational graph in a backwards direction—from the output loss back to the input parameters—multiplying matrices of partial derivatives at each step to determine how parameters affect the final output.

Backpropagation

When evaluating a vector-valued function, its underlying computational dependency graph is traversed in a forward direction, from the input variables to the output. This forward pass calculates the function's intermediate and final values before any gradients are computed.

Forward Graph Traversal

To compute the gradients of a vector-valued function, its computational dependency graph is traversed in a backwards direction, tracing from the output back to the inputs. During this backward pass, matrices of partial derivatives are multiplied along the paths to determine how changes in the input variables affect the final output.

Backward Graph Traversal

One of the major benefits of automatic differentiation is its ability to compute gradients even when the sequence of operations depends on dynamic control flow, such as Python if statements, while loops, and arbitrary function calls. Since the exact path of computation is not known a priori, the computational graph is realized dynamically only during the forward evaluation on a specific input. Once this specific graph is constructed, deep learning frameworks can execute the backward pass to accurately calculate the gradient of the resulting variable.

Automatic Differentiation with Dynamic Control Flow

Plotting a computational graph helps visualize the mathematical dependencies between variables and operators during the forward propagation of a neural network. In these visual representations, variables are typically denoted by squares and operators by circles. Arrows are used to indicate the direction of data flow; in a forward pass, this flow typically proceeds from the network inputs (often positioned at the lower-left) toward the outputs (at the upper-right).

Computational Graph of Forward Propagation

A computational graph is a directed dependency graph where nodes represent mathematical operations or variables, mapping the functional dependency from inputs to outputs. When evaluating a vector-valued function, this graph is traversed in a forward direction. Conversely, to compute gradients automatically, the graph is traced in a backwards direction from output to input, which involves multiplying matrices of partial derivatives along the paths.

University of Michigan - Ann Arbor

Claude

San Jose State University

While single-variable calculus deals with functions of one variable, deep learning often requires working with functions of many variables, known as multivariate functions. A multivariate function takes an $$n$$-dimensional input, representing multiple parameters, and evaluates them to produce an output. This is typically written as $$y = f(x_1, x_2, \ldots, x_n)$$, where the variables $$x_1$$ through $$x_n$$ jointly determine the value of the function.

Multivariate Function

The derivative of a vector $$\mathbf{u}$$ (consisting of $$m$$ variables) with respect to a vector $$\mathbf{x}$$ (consisting of $$n$$ variables) is an $$n 	imes m$$ matrix, commonly denoted as $$\mathbf{A}$$. This matrix compiles all the individual partial derivatives of the components of $$\mathbf{u}$$ with respect to the components of $$\mathbf{x}$$.

Derivative of a Vector with Respect to a Vector

A neural network is typically structured with an input layer, one or more hidden layers, and an output layer. Its design is historically inspired by biological neurons: dendrites are modeled as inputs $$x_i$$, synapses as synaptic weights $$w_i$$, and the nucleus as an aggregator that computes a weighted sum $$y = \sum_i x_i w_i + b$$, potentially followed by a nonlinear activation function $$\sigma(y)$$. The resulting signal is then transmitted to the output, analogous to a biological axon. A positive weight reflects an excitatory connection, while a negative value signifies an inhibitory connection.

Neural Network Representation

https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning

Neural Networks and Deep Learning

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

Computational Graph

The partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^	extrm{th}$$ parameter $$x_i$$ measures how the function changes as $$x_i$$ increases while treating all other variables as constants. Formally, it is defined as the limit: $$\frac{\partial y}{\partial x_i} = \lim_{h ightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$.

Partial Derivative

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Jacobian Matrix

It receives input from some other nodes, or from an external source and computes an output. Each input has an associated weight (w), which is assigned on the basis of its relative importance to other inputs. Below is an example of a neuron for logistic regression.

Singular neuron of NN

Linear regression can be conceptualized as a single-layer, fully connected neural network. In this representation, each given input feature $$x_1, \ldots, x_d$$ corresponds to an input neuron. Since the goal is to predict a single numerical value, all input neurons are directly connected to a single computed output neuron $$o_1$$. The total number of inputs $$d$$ is referred to as the feature dimensionality in the input layer.

Learn Before

Related

Learn After