After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


University of Michigan - Ann Arbor

Models that are too simple that don't even do well on the training data, are said to **underfit** and are not likely to generalize well to new examples.

Underfitting a supervised statistical model

Increasing the flexibility of a supervised statistical model may result in overfitting, such that the trained model $\hat{f}$ gets prone to the noise or errors in the observations (training data) rather than approximating the true $f$.
So, when choosing a functional form and training a supervised statistical model, we should always consider a trade-off between the flexibility and overfitting of $\hat{f}$.

Models that re too complex for the amount of training data available are said to **overfit** and are not likely to generalize well to new examples.

Overfitting a supervised statistical model

In Ridge regression the coefficients and bias are learned using the same least-square criterion, but it adds a penalty for large variations in coefficients; i.e., coefficients are found by minimizing a tuning parameter - which controls the strength of the penalty term.
Once the parameters are learned, the ridge regression prediction formula is the same as OLS.
Ridge regression uses L2 regularization that minimizes the sum of square of coefficients and the influence of the regularization term is controlled by the $\alpha$ parameter. Higher $\alpha$ means more regularization and simpler models.
Use Ridge regression when the number of predictor variables is greater than the number of observations. Below is the formula found in our textbook. 
$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) =
\sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2.
$$ 

Note: Ridge Regression is sensitive to scales of variable. Therefore, we usually standardize the predictors before applying Ridge Regression.

Ridge Regression

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

The left plot shows fitting a straight line to the data. This is not a very good fit. So, it is underfitting the data and we call it high bias. On the opposite end, if you fit an incredibly complex classifier, maybe you can fit the data perfectly, but that doesn't look like a great fit either. So there's a classifier of high variance and this is overfitting the data. And there might be some classifier in between, with a medium level of complexity, that maybe fits it correctly.

Overfitting/Underfitting vs. Bias/Variance in Supervised Machine Learning

Which of the following would be the best choice for the next ridge regression model you train?

One common way to detirmine whether or not your model is overfitting is by analyzing the accuracy of the model's performance on the training data and test data. Your model may be overefit if you have very high accuracy on the training data, and then much lower accuracy on the testing data.

How to tell when your model is overfitting

1. Cross-validation
2. Train with more data.
3. Remove features.
4. Early stopping.
5. Regularization.
6. Ensembling.

How to avoid overfitting

While guarantees from classical learning theory (such as bounds based on the Vapnik-Chervonenkis (VC) dimension or Rademacher complexity) can be conservative even for classical models, they appear powerless to explain why deep neural networks generalize. For classification problems, deep models are typically expressive enough to perfectly fit arbitrary labels even for datasets consisting of millions of examples. In the classical picture, such extreme model complexity—even when utilizing familiar methods like $$\ell_2$$ regularization—should lead to severe overfitting. Paradoxically, despite perfectly fitting the training data with zero training error, these highly expressive models often generalize remarkably well to unseen data, contradicting traditional complexity-based generalization bounds.

Generalization Paradox in Deep Learning

1. Bias-variance trade-off: as $$\lambda$$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

When the number of variables p is almost as large as the number of observations n, the least squares estimates will be extremely variable. And if p > n, then the least squares estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance.

Hence, ridge regression works best in situations where the least squares estimates have high variance.

2. Substantial computational advantages over best subset selection.

Why does Ridge Regression Improve Over Least Squres?

```python
#library imports
from sklearn.linear_model import Ridge
import numpy as np 
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

#Get Dataset and set X and y data.
(X_data, y_data) = load_dataset()

#splitting
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, random_state = 0)

#creating a model with any alpha value for regularization.
ridge = Ridge(alpha=20.0)

#using model (fit method).
ridge.fit(X_train, y_train)

#results and attributes
print('ridge regression linear model intercept: {}'
     .format(ridge.intercept_))
print('ridge regression linear model coeff:\n{}'
     .format(ridge.coef_))
print('R-squared score (training): {:.3f}'
     .format(ridge.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
     .format(ridge.score(X_test, y_test)))
print('Number of non-zero features: {}'
     .format(np.sum(ridge.coef_ != 0)))
```

Simple Ridge Regression Example in Python

In general, one might expect the Lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size.

However, the number of predictors that is related to the response if never known a priori for real data sets. A technique such as cross-validation can be used to determine which approach is better on a particular data set.

Ridge Regression vs Lasso Regression

The influence of the regularization term is controlled by the $\lambda$

Learn Before

Related