After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


University of Michigan - Ann Arbor

Google

Models that are too simple that don't even do well on the training data, are said to **underfit** and are not likely to generalize well to new examples.

Underfitting a supervised statistical model

Increasing the flexibility of a supervised statistical model can lead to overfitting, where the trained model $$\hat{f}$$ fits the noise or random errors in the training observations rather than approximating the true underlying function $$f$$. Consequently, when selecting a model class and training a supervised statistical model, one must balance model flexibility with the risk of overfitting. Models that are too complex relative to the size of the training dataset are highly susceptible to overfitting and typically fail to generalize well to new, unseen examples.

Overfitting a supervised statistical model

In ridge regression, the model coefficients and bias are learned using the least-squares criterion with an added penalty for large coefficients. The parameters are found by minimizing an objective function that includes a tuning parameter (often denoted as $$\lambda$$ or $$\alpha$$) to control the strength of this penalty term. Once the parameters are learned, the prediction formula is the same as in ordinary least squares (OLS).

Ridge regression uses $$L_2$$ regularization (weight decay), which adds the sum of the squared coefficients to the residual sum of squares (RSS):

$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) = \sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2$$

A higher tuning parameter ($$\lambda$$ or $$\alpha$$) results in more regularization and a simpler model. Ridge regression is especially useful when the number of predictor variables is greater than the number of observations. Because the penalty term is sensitive to the scale of the variables, predictors should usually be standardized before applying ridge regression.

Ridge Regression

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

The left plot shows fitting a straight line to the data. This is not a very good fit. So, it is underfitting the data and we call it high bias. On the opposite end, if you fit an incredibly complex classifier, maybe you can fit the data perfectly, but that doesn't look like a great fit either. So there's a classifier of high variance and this is overfitting the data. And there might be some classifier in between, with a medium level of complexity, that maybe fits it correctly.

Overfitting/Underfitting vs. Bias/Variance in Supervised Machine Learning

Which of the following would be the best choice for the next ridge regression model you train?

1. Cross-validation
2. Train with more data.
3. Remove features.
4. Early stopping.
5. Regularization.
6. Ensembling.

How to avoid overfitting

While guarantees from classical learning theory (such as bounds based on the Vapnik-Chervonenkis (VC) dimension or Rademacher complexity) can be conservative even for classical models, they appear powerless to explain why deep neural networks generalize. For classification problems, deep models are typically expressive enough to perfectly fit arbitrary labels even for datasets consisting of millions of examples. In the classical picture, such extreme model complexity—even when utilizing familiar methods like $$\ell_2$$ regularization—should lead to severe overfitting. Paradoxically, despite perfectly fitting the training data with zero training error, these highly expressive models often generalize remarkably well to unseen data, contradicting traditional complexity-based generalization bounds.

Generalization Paradox in Deep Learning

A common method to determine whether a supervised statistical model is overfitting is to compare its performance on the training data against unseen test data. If the model achieves very high accuracy on the training data but significantly lower accuracy on the testing data, it is likely overfitting.

Identifying Overfitting via Training and Test Performance

1. Bias-variance trade-off: as $$\lambda$$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

When the number of variables p is almost as large as the number of observations n, the least squares estimates will be extremely variable. And if p > n, then the least squares estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance.

Hence, ridge regression works best in situations where the least squares estimates have high variance.

2. Substantial computational advantages over best subset selection.

Why does Ridge Regression Improve Over Least Squres?

```python
#library imports
from sklearn.linear_model import Ridge
import numpy as np 
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

#Get Dataset and set X and y data.
(X_data, y_data) = load_dataset()

#splitting
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, random_state = 0)

#creating a model with any alpha value for regularization.
ridge = Ridge(alpha=20.0)

#using model (fit method).
ridge.fit(X_train, y_train)

#results and attributes
print('ridge regression linear model intercept: {}'
     .format(ridge.intercept_))
print('ridge regression linear model coeff:\n{}'
     .format(ridge.coef_))
print('R-squared score (training): {:.3f}'
     .format(ridge.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
     .format(ridge.score(X_test, y_test)))
print('Number of non-zero features: {}'
     .format(np.sum(ridge.coef_ != 0)))
```

Simple Ridge Regression Example in Python

In general, one might expect the Lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size.

However, the number of predictors that is related to the response if never known a priori for real data sets. A technique such as cross-validation can be used to determine which approach is better on a particular data set.

Ridge Regression vs Lasso Regression

In ridge regression, the tuning parameter (often denoted as $$\lambda$$) controls the influence of the regularization term and directly impacts the model's tendency to overfit. A higher $$\lambda$$ applies a stronger penalty to the model coefficients, resulting in a simpler, less flexible model. This increased regularization helps prevent overfitting by reducing the model's variance, although it may increase bias.

Learn Before

Related