University of Michigan - Ann Arbor

Weight decay, commonly known as $$\ell_2$$ regularization, is a widely used technique for regularizing parametric machine learning models. Instead of directly manipulating the number of parameters, weight decay operates by restricting the values that the parameters can take. The technique is motivated by the intuition that the simplest function is $$f = 0$$, and the complexity of a linear function, such as $$f(\mathbf{x}) = \mathbf{w}^	op \mathbf{x}$$, can be measured by the distance of its parameters from zero.

Weight Decay

In ridge regression, the model coefficients and bias are learned using the least-squares criterion with an added penalty for large coefficients. The parameters are found by minimizing an objective function that includes a tuning parameter (often denoted as $$\lambda$$ or $$\alpha$$) to control the strength of this penalty term. Once the parameters are learned, the prediction formula is the same as in ordinary least squares (OLS).

Ridge regression uses $$L_2$$ regularization (weight decay), which adds the sum of the squared coefficients to the residual sum of squares (RSS):

$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) = \sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2$$

A higher tuning parameter ($$\lambda$$ or $$\alpha$$) results in more regularization and a simpler model. Ridge regression is especially useful when the number of predictor variables is greater than the number of observations. Because the penalty term is sensitive to the scale of the variables, predictors should usually be standardized before applying ridge regression.

Ridge Regression

What is weight decay?

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

The regularization constant, $$\lambda$$, is a nonnegative hyperparameter that characterizes the trade-off between standard prediction loss and a regularization penalty. It is typically fit using validation data. When $$\lambda = 0$$, the original loss function is recovered, while larger values of $$\lambda$$ constrain the model's weights more considerably. For example, in weight decay ($$\ell_2$$ regularization), it modifies the objective to $$L(\mathbf{w}, b) + \frac{\lambda}{2} \|\mathbf{w}\|^2$$, where the penalty term is conventionally divided by 2 so that the constant cancels out gracefully when taking the derivative.

Regularization Constant

In deep learning, typical strengths of weight decay (such as $$\ell_2$$ regularization) are usually insufficient to prevent highly parameterized networks from fully interpolating the training data. While these regularizers were classically thought to restrict models from fitting arbitrary labels, their effectiveness in deep architectures is often contingent on being paired with an early stopping criterion.

Insufficiency of Weight Decay for Preventing Interpolation

In deep learning, the _L2_ norm and _Frobenius_ norm are often used interchangeably, though they differ slightly. The _L2_ norm (Euclidean norm) applies to vectors and is the special case $$p=2$$ of the $$L_p$$ norm: $$L_2 = \sqrt{\textstyle\sum_{i=1}^n x_{i}^{2}}$$. The _Frobenius_ norm measures the analogous quantity for matrices: the square root of the sum of the squared magnitudes of all matrix elements. Because neural network weight matrices are regularized this way, this matrix norm is commonly (though loosely) referred to as the _L2_ norm in neural network literature, even though it is technically the _Frobenius_ norm.

Learn Before

Related