In the implementation of batch normalization, a momentum parameter is utilized to govern the exponential moving average aggregation of past mean and variance estimates during training. These aggregated dataset statistics are then relied upon during the model's prediction mode. Although it shares the name, this momentum parameter is a misnomer and has absolutely nothing to do with the momentum term used to accelerate optimization algorithms; it is simply the convention used in high-level APIs to describe the smoothing factor for the tracked moving averages.

Claude

Batch normalization is applied to individual layers by standardizing the inputs based on the statistics of the current minibatch $$\mathcal{B}$$. For an input $$\mathbf{x} \in \mathcal{B}$$, the batch normalization $$\textrm{BN}$$ is defined as:

$$ \textrm{BN}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \hat{\boldsymbol{\mu}}_{\mathcal{B}}}{\hat{\boldsymbol{\sigma}}_{\mathcal{B}}} + \boldsymbol{\beta} $$

Here, $$\hat{\boldsymbol{\mu}}_{\mathcal{B}} = \frac{1}{|\mathcal{B}|} \sum_{\mathbf{x} \in \mathcal{B}} \mathbf{x}$$ is the sample mean, and $$\hat{\boldsymbol{\sigma}}_{\mathcal{B}}^2 = \frac{1}{|\mathcal{B}|} \sum_{\mathbf{x} \in \mathcal{B}} (\mathbf{x} - \hat{\boldsymbol{\mu}}_{\mathcal{B}})^2 + \epsilon$$ is the sample variance with a small constant $$\epsilon > 0$$ added for numerical stability to prevent division by zero. The parameters $$\boldsymbol{\gamma}$$ (scale parameter) and $$\boldsymbol{\beta}$$ (shift parameter) are learned during training to recover the degrees of freedom lost due to standardization.

Batch Normalization Formula

Dive into Deep Learning

For t=1,..., n mini-batches,
         Compute forward propagation on $X^{\{i\}}$
                 In each hidden layer, use Batch Norm to replace $z^{(l)}$ with $\tilde{z}^{(l)}$
          Use back propagation to compute $dW^{(l)}, d\gamma^{(l)},  d\beta^{(l)}$
          Update parameters:
                   $W^{(l)} = W^{(l)} - dW^{(l)}$
                   $\gamma^{(l)} = \gamma^{(l)} - d\gamma^{(l)}$
                   $\beta^{(l)} = \beta^{(l)} - d\beta^{(l)}$

Gradient Descent Batch Norm in Deep Learning Implementation

Batch normalization layers function differently during the prediction (or test) mode compared to the training mode. During training, the normalization is based on the statistics of the current minibatch, which change with every model update. Once the model is fully trained, however, the exact means and variances of each layer's variables are calculated based on the entire dataset. In prediction mode, the model uses these fixed dataset-wide statistics to ensure that the same input is classified consistently, regardless of the batch it resides in.

Batch Normalization in Prediction Mode

In convolutional layers, batch normalization is typically applied after the convolution operation but before the nonlinear activation function. To preserve the translation invariance of convolutions, the normalization is executed on a per-channel basis simultaneously across all spatial locations. For a minibatch containing $$m$$ examples and an output feature map with height $$p$$ and width $$q$$, the mean and variance are calculated over all $$m \cdot p \cdot q$$ elements for each individual channel. Consequently, each channel utilizes the same scalar scale and shift parameters to normalize values at every spatial location.

Batch Normalization in Convolutional Layers

The programmatic mathematical implementation of batch normalization distinguishes between training and prediction modes. During training mode, it calculates the sample mean and variance over the current minibatch along the appropriate dimensions—the feature dimension for fully connected layers or the channel dimension for convolutional layers. It then normalizes the input using these statistics and updates an exponential moving average of the mean and variance. In prediction mode, it bypasses the batch statistics calculation and directly uses the tracked moving averages. Finally, it applies the learnable scale ($$\boldsymbol{\gamma}$$) and shift ($$\boldsymbol{\beta}$$) parameters to the normalized output.

Learn Before

Related