University of Michigan - Ann Arbor

Google

1. Validation set
2. Leave-One-Out Cross-Validation
3. k-fold Cross-Validation

Cross-Validation resampling methods

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

In Leave-One-Out Cross-Validation (LOOCV), if a dataset has $$n$$ observations, the model is trained on $$n-1$$ data points and a prediction is made on the single left-out observation. This process is repeated $$n$$ times so that each data point is used as the validation set exactly once. The model's overall performance is evaluated by averaging the performance across all $$n$$ predictions.

Leave-One-Out Cross-Validation

To perform $$K$$-fold cross-validation on a dataset with $$n$$ observations, the training set is separated into $$K$$ sets by randomly sampling $$n/K$$ observations from the dataset into each fold. The model of interest is then trained on $$K-1$$ folds and validated on the left-out fold; that is, once the model is trained, we use it to make predictions on the group that was left out of training. This process repeats $$K$$ times. To evaluate the model's overall performance, we calculate the average validation error across all $$K$$ tests. $$K$$-fold cross-validation provides a robust estimate of the empirical testing error of a model and is particularly useful for selecting model designs and adjusting hyperparameters.

K-Fold Cross-Validation

The advantage of k-fold CV is that it often gives more accurate estimates of the test error rate than does LOOCV. This has to do with a bias-variance trade-off.
The validation set approach can lead to overestimates of the test error rate, since in this approach the training set used to fit the statistical learning method contains only half the observations of the entire data set ( it means the model does not have enough information about the data set.)
LOOCV will give approximately unbiased estimates of the test error since each training set contains n − 1 observation, which is almost as many as the number of observations in the full data set.
K-fold CV for, say, k = 5 or k = 10 will lead to an intermediate level of
bias, since each training set contains (k − 1)n/k observations—fewer than
in the LOOCV approach, but substantially more than in the validation set
approach.

Note:
LOOCV has a higher variance than does K-fold CV with k < n.

Bias-Variance Trade Off Comparison

Stratified folds each contain a proportion of classes that matches the overal dataset. Now, all cclasses will be fairly represented in the test set.

Stratified Cross-Validation

Which of the following is true of cross-validation?

A validation dataset (or validation set) is a separate partition of data used to assess the quality of a machine learning model's solution. After training a model iteratively on the training data, its predictive performance is evaluated on this distinct validation dataset to measure how well it has learned the underlying patterns. This evaluation is critical for automatically tuning hyperparameters without relying on the training set, ensuring the model generalizes appropriately to unseen data.

Learn Before

Related