In general, one might expect the Lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size.

However, the number of predictors that is related to the response if never known a priori for real data sets. A technique such as cross-validation can be used to determine which approach is better on a particular data set.

University of Michigan - Ann Arbor

In Ridge regression the coefficients and bias are learned using the same least-square criterion, but it adds a penalty for large variations in coefficients; i.e., coefficients are found by minimizing a tuning parameter - which controls the strength of the penalty term.
Once the parameters are learned, the ridge regression prediction formula is the same as OLS.
Ridge regression uses L2 regularization that minimizes the sum of square of coefficients and the influence of the regularization term is controlled by the $\alpha$ parameter. Higher $\alpha$ means more regularization and simpler models.
Use Ridge regression when the number of predictor variables is greater than the number of observations. Below is the formula found in our textbook. 
$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) =
\sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2.
$$ 

Note: Ridge Regression is sensitive to scales of variable. Therefore, we usually standardize the predictors before applying Ridge Regression.

Ridge Regression

Lasso Regression is similar to ridge regression, but instead we use the L1 regularization penalty.
This has the effect of setting some coefficient estimates to exactly zero for the least influential variables, which leads to models only including a subset of variables. This is called a space solution, which is a kind of feature selection.
The parameter $\alpha$ controls the amount of L1 regularization (default = 1.0).
The prediction formula is the same as OLS.

$$RSS_{LASSO}(\beta_{j}, \beta_{0}) =
\sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert=RSS+\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert.
$$ 

Lasso Regression

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

1. Bias-variance trade-off: as $$\lambda$$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

When the number of variables p is almost as large as the number of observations n, the least squares estimates will be extremely variable. And if p > n, then the least squares estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance.

Hence, ridge regression works best in situations where the least squares estimates have high variance.

2. Substantial computational advantages over best subset selection.

Why does Ridge Regression Improve Over Least Squres?

```python
#library imports
from sklearn.linear_model import Ridge
import numpy as np 
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

#Get Dataset and set X and y data.
(X_data, y_data) = load_dataset()

#splitting
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, random_state = 0)

#creating a model with any alpha value for regularization.
ridge = Ridge(alpha=20.0)

#using model (fit method).
ridge.fit(X_train, y_train)

#results and attributes
print('ridge regression linear model intercept: {}'
     .format(ridge.intercept_))
print('ridge regression linear model coeff:\n{}'
     .format(ridge.coef_))
print('R-squared score (training): {:.3f}'
     .format(ridge.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
     .format(ridge.score(X_test, y_test)))
print('Number of non-zero features: {}'
     .format(np.sum(ridge.coef_ != 0)))
```

Simple Ridge Regression Example in Python

Ridge Regression vs Lasso Regression

After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


Which of the following would be the best choice for the next ridge regression model you train?

The influence of the regularization term is controlled by the $\lambda$

Learn Before

Related