Divides the training set into two parts - a training set and a validation set (also known as the “holdout set”). A model is made to fit the training set, and then the validation set is used to test the accuracy of the model because the model is blind to it. This gives an estimate of the test error of the model and can be useful in determining the best model from a series of models using different hyperparameters.

Cross-Validation for resampling

This approach uses information criteria to calculate an expected score out-of-sample. It constructs a theoretical estimate of the relative out-of-sample KL divergence. 

The best-known information criterion is the Akaike Information Criterion (AIC). This tells us is the dimensionality of the posterior distribution is a natural measure of the model's overfitting tendency.

However,  AIC is an approximation only reliable when:
- The prior are flat or overwhelmed by the likelihood 
- The posterior distribution is approximately multivariate Gaussian 
- The sample size (N) is much greater than the number of parameters (k)


Information Criteria

Two strategies:
- Cross-Validation
- Information Criteria

These stategies guess how well model will predict data.

Arizona State University

- Compare models through cross validation and information criteria to determine the predictive accuracy and tendency for overfitting, in addition to highlighting any observations that are exerting significant influence to the model 
- Overfitting: simple models may lead to more accurate predictions than more complicated models

Model comparison and prediction

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

A "skeptical" prior that, when tuned properly,  slows the rate of learning from the sample. This reduces overfitting while allowing the model to learn the regular features of the sample.



Regularization Priors

Predictive Accuracy

Need to identify goal:
1.  Making Predictions
2. Inferring Cause

Learn Before

Related

Learn After