Training a neural network proceeds as an iterative loop: (1) forward propagation computes the network's output from the current weights and biases, (2) a loss function measures the error between that output and the target, (3) backpropagation computes the gradient of the loss with respect to each weight and bias, and (4) an optimization step (e.g., gradient descent) updates the parameters using those gradients. This loop repeats over the training data until the loss converges to an acceptably low value.

Google

Training a neural network is the process of adjusting its weights and biases so that the network's outputs come to match the desired targets as closely as possible. Starting from an initial set of parameters, the network computes an output via forward propagation, compares that output to the target using a loss (error) function, and then updates the weights and biases based on the resulting error signal, typically via backpropagation combined with a gradient-based optimization step. This iterative fine-tuning of weights and biases from the training data, repeated until the network's performance converges, is what is meant by "training" a neural network.

Training Neural Networks

During neural network training, forward and backward propagation are fundamentally interdependent. Forward propagation depends on the current model parameters, which are updated by the optimization algorithm using gradients calculated during the most recent backpropagation step. Conversely, backpropagation traverses the computational graph in reverse and depends on the intermediate variables that were computed and stored during the forward propagation pass. For example, computing the regularization term during forward propagation depends on the current values of model parameters $$\mathbf{W}^{(1)}$$ and $$\mathbf{W}^{(2)}$$, which are given by the optimization algorithm according to backpropagation in the most recent iteration. On the other hand, the gradient calculation during backpropagation depends on the current value of the hidden layer output $$\mathbf{h}$$, which is given by forward propagation.

Learn Before

Related