Regularization is a modification made to a learning algorithm with the goal of reducing generalization error, but not training error.

Linear Regression is useful for modeling outputs whose relation to their inputs is linear. However, linear regression performs poorly when trying to model non-linear relationships. We can use regularization to help learning algorithms select certain models over others, particularly to avoid overfitting. 

Regularization

The dashed curve is meant to regularize. Since there is more probability around 0, the estimate of 0 shrinks. The other 2 curves are narrower priors and therefore are more skeptical of values further from 0. The thin curve has a standard deviation of 0.5 and the thick curve has a standard distribution of 0.2. 

Reducing Overfitting with Different Strengths

A "skeptical" prior that, when tuned properly,  slows the rate of learning from the sample. This reduces overfitting while allowing the model to learn the regular features of the sample.



Arizona State University

- Compare models through cross validation and information criteria to determine the predictive accuracy and tendency for overfitting, in addition to highlighting any observations that are exerting significant influence to the model 
- Overfitting: simple models may lead to more accurate predictions than more complicated models

Model comparison and prediction

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

Regularization Priors

Two strategies:
- Cross-Validation
- Information Criteria

These stategies guess how well model will predict data.

Predictive Accuracy

Need to identify goal:
1.  Making Predictions
2. Inferring Cause

Learn Before

Related

Learn After