University of Michigan - Ann Arbor

1. Validation set
2. Leave-One-Out Cross-Validation
3. k-fold Cross-Validation

Cross-Validation resampling methods

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If we have $n$ data points in our training set, we train the model on $n-1$ data points. Then, we make a prediction on the observation that we left out. We repeat this process repeats $n$ times, and we will average the performance of our model for each of the predictions that we make. This average will give us model performance.

Leave-One-Out Cross-Validation

To perform K-fold CV on a dataset with $$n$$ observations, the training set is separated into $$k$$ sets by randomly sampling $$n/k$$ observations from the dataset into each fold. 
The model of interest is then trained on $$k−1$$ folds and validated on the left-out fold, i.e., once the model is trained, we use it to make predictions on the group that was left out of training. This process repeats $n$ times. To get the model performance, we average the results from all of the tests. K-Fold CV proves a good measure of the empirical testing error of a model. 


K-Fold Cross-Validation

The advantage of k-fold CV is that it often gives more accurate estimates of the test error rate than does LOOCV. This has to do with a bias-variance trade-off.
The validation set approach can lead to overestimates of the test error rate, since in this approach the training set used to fit the statistical learning method contains only half the observations of the entire data set ( it means the model does not have enough information about the data set.)
LOOCV will give approximately unbiased estimates of the test error since each training set contains n − 1 observation, which is almost as many as the number of observations in the full data set.
K-fold CV for, say, k = 5 or k = 10 will lead to an intermediate level of
bias, since each training set contains (k − 1)n/k observations—fewer than
in the LOOCV approach, but substantially more than in the validation set
approach.

Note:
LOOCV has a higher variance than does K-fold CV with k < n.

Bias-Variance Trade Off Comparison

Stratified folds each contain a proportion of classes that matches the overal dataset. Now, all cclasses will be fairly represented in the test set.

Stratified Cross-Validation

Which of the following is true of cross-validation?

A validation dataset (or validation set) is a separate partition of data used to assess the quality of a machine learning model's solution. After training a model iteratively on the training data, its predictive performance is evaluated on this distinct validation dataset to measure how well it has learned the underlying patterns. This evaluation is critical for automatically tuning hyperparameters without relying on the training set, ensuring the model generalizes appropriately to unseen data.

Learn Before

Related