In ridge regression, the tuning parameter (often denoted as $$\lambda$$) controls the influence of the regularization term and directly impacts the model's tendency to overfit. A higher $$\lambda$$ applies a stronger penalty to the model coefficients, resulting in a simpler, less flexible model. This increased regularization helps prevent overfitting by reducing the model's variance, although it may increase bias.

University of Michigan - Ann Arbor

Google

In ridge regression, the model coefficients and bias are learned using the least-squares criterion with an added penalty for large coefficients. The parameters are found by minimizing an objective function that includes a tuning parameter (often denoted as $$\lambda$$ or $$\alpha$$) to control the strength of this penalty term. Once the parameters are learned, the prediction formula is the same as in ordinary least squares (OLS).

Ridge regression uses $$L_2$$ regularization (weight decay), which adds the sum of the squared coefficients to the residual sum of squares (RSS):

$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) = \sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2$$

A higher tuning parameter ($$\lambda$$ or $$\alpha$$) results in more regularization and a simpler model. Ridge regression is especially useful when the number of predictor variables is greater than the number of observations. Because the penalty term is sensitive to the scale of the variables, predictors should usually be standardized before applying ridge regression.

Ridge Regression

1. Bias-variance trade-off: as $$\lambda$$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

When the number of variables p is almost as large as the number of observations n, the least squares estimates will be extremely variable. And if p > n, then the least squares estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance.

Hence, ridge regression works best in situations where the least squares estimates have high variance.

2. Substantial computational advantages over best subset selection.

Why does Ridge Regression Improve Over Least Squres?

```python
#library imports
from sklearn.linear_model import Ridge
import numpy as np 
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

#Get Dataset and set X and y data.
(X_data, y_data) = load_dataset()

#splitting
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, random_state = 0)

#creating a model with any alpha value for regularization.
ridge = Ridge(alpha=20.0)

#using model (fit method).
ridge.fit(X_train, y_train)

#results and attributes
print('ridge regression linear model intercept: {}'
     .format(ridge.intercept_))
print('ridge regression linear model coeff:\n{}'
     .format(ridge.coef_))
print('R-squared score (training): {:.3f}'
     .format(ridge.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
     .format(ridge.score(X_test, y_test)))
print('Number of non-zero features: {}'
     .format(np.sum(ridge.coef_ != 0)))
```

Simple Ridge Regression Example in Python

In general, one might expect the Lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size.

However, the number of predictors that is related to the response if never known a priori for real data sets. A technique such as cross-validation can be used to determine which approach is better on a particular data set.

Ridge Regression vs Lasso Regression

After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


Learn Before

Related