Sometimes when n is extremely large, performing LOOCV may pose computational problems. However, performing a 10-fold CV requires training the model only ten times, which reduces the computational complexity.

K-Fold Cross-Validation is less expensive than LOOCV

K-Fold CV is usually the method of choice for CV, and functions as a sort of generalization of LOOCV, because:
1. It is much less computationally expensive than LOOCV.
2. It minimizes the **Bias-Variance Tradeoff**.

University of Michigan - Ann Arbor

If we have $n$ data points in our training set, we train the model on $n-1$ data points. Then, we make a prediction on the observation that we left out. We repeat this process repeats $n$ times, and we will average the performance of our model for each of the predictions that we make. This average will give us model performance.

Leave-One-Out Cross-Validation

To perform K-fold CV on a dataset with $$n$$ observations, the training set is separated into $$k$$ sets by randomly sampling $$n/k$$ observations from the dataset into each fold. 
The model of interest is then trained on $$k−1$$ folds and validated on the left-out fold, i.e., once the model is trained, we use it to make predictions on the group that was left out of training. This process repeats $n$ times. To get the model performance, we average the results from all of the tests. K-Fold CV proves a good measure of the empirical testing error of a model. 


K-Fold Cross-Validation

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

K-Fold Cross-Validation vs. Leave-One-Out Cross-Validation

When selecting $K$ for K-Fold CV, we should consider a bias-variance trade-off. K-Fold CV provides $K$ estimates of the model error. The mean of the errors indicates the model's bias. A lower mean value indicates a higher accuracy of the model. The model variance is calculated based on the standard deviation of the errors. A lower variance indicates a low variation in the model performance. Usually, it is difficult to reduce both model bias and variance.

K-Fold Cross-Validation Bias-Variance Tradeoff

$$5 \leq K \leq 10$$ is commonly used in Macine Learning. A too-small $K$ may lead to high bias, and a too-large $K$ may result in high variance. 

Learn Before

Related

Learn After