As a model becomes more flexible, it usually fits the training data more closely. That tends to reduce bias but increase variance, so the test error often falls at first and then rises again. The best model complexity is typically the one that gives the lowest validation error.

University of Michigan - Ann Arbor

When selecting $$K$$ for $$K$$-Fold Cross-Validation, we must consider the bias-variance tradeoff. $$K$$-Fold Cross-Validation provides $$K$$ estimates of the model error. The mean of these errors indicates the model's bias, where a lower mean value implies higher accuracy. The model variance is determined by the standard deviation of these errors, where a lower variance indicates less fluctuation in model performance. Usually, it is difficult to reduce both model bias and variance simultaneously.

K-Fold Cross-Validation Bias-Variance Tradeoff

Understanding the Bias-Variance Tradeoff.  (2012, June). Retrieved from http://scott.fortmann-roe.com/docs/BiasVariance.html

Understanding the Bias-Variance Tradeoff

Prediction Error as Model Flexibility Changes

A useful way to picture the bias-variance tradeoff is to imagine three regions of model complexity. Very simple models tend to miss important patterns, which gives them high bias and low variance. Very flexible models can fit noise in the training data, which gives them low bias and high variance. The practical goal is to choose a middle ground that predicts well on new data.

Learn Before

Related