The memory footprint required to retain intermediate values during neural network training is roughly proportional to the number of network layers and the batch size. Because these variables must be stored until backpropagation concludes, training deeper networks with larger batch sizes leads to a substantial increase in memory consumption, making the system more susceptible to out-of-memory errors.

Memory Footprint of Neural Network Training

To avoid duplicate calculations, backpropagation reuses the intermediate values that were computed during the forward propagation pass. As a result, all intermediate variables must be retained in memory until backpropagation is completely finished. This retention requirement is a major reason why training a neural network demands significantly more memory than plain prediction (inference), where intermediate values can be immediately discarded.

Claude

During neural network training, forward and backward propagation are fundamentally interdependent. Forward propagation depends on the current model parameters, which are updated by the optimization algorithm using gradients calculated during the most recent backpropagation step. Conversely, backpropagation traverses the computational graph in reverse and depends on the intermediate variables that were computed and stored during the forward propagation pass. For example, computing the regularization term during forward propagation depends on the current values of model parameters $$\mathbf{W}^{(1)}$$ and $$\mathbf{W}^{(2)}$$, which are given by the optimization algorithm according to backpropagation in the most recent iteration. On the other hand, the gradient calculation during backpropagation depends on the current value of the hidden layer output $$\mathbf{h}$$, which is given by forward propagation.

Learn Before

Related

Learn After