
1. When different observations are included in and excluded the validation set, the estimated test error rate could be highly variable.
2. Fitting the model with only part of all the observations might lead to bad performance.

Potential drawbacks of Validation Set Approach

After training the model on the training set, the validation set is used to tune parameters. The test then set is used to see how the model can be generalized to new data.

Difference between test set and validation set

In machine learning, generalization refers to the formidable challenge of finding a model whose parameters lead to accurate predictions on previously unseen data. While deep learning optimization algorithms can often easily find parameters that minimize the loss on the training set, ensuring that those learned parameters adapt well and maintain high predictive accuracy on new, independent datasets is the ultimate objective of the learning process.

Generalization

While the training loop is responsible for optimizing a model's parameters, a validation dataset is crucial for measuring the model's quality on unseen data. A standard practice is to pass a validation dataloader through the model once during each epoch. By evaluating the model's performance without updating its parameters—often in a no-gradient context—practitioners can monitor generalization and detect overfitting during the iterative training process.

Validation Dataset Model Evaluation

Highly expressive models, such as deep neural networks, possess the capacity to perfectly fit even arbitrarily assigned training labels. For these powerful models, achieving a low training error is insufficient to guarantee a low generalization error, although it does not inherently imply a high generalization error either. Because training error provides an unreliable signal about true predictive performance in these cases, practitioners must rely heavily on holdout data to certify generalization after training, calculating what is known as the validation error.

Validation Error for Expressive Models

Because the strict boundaries between validation and test data are often blurred in practice due to data reuse, it is important to distinguish between reported validation accuracy and theoretical true test set accuracy. When a true, completely unseen test set is not available or has been evaluated multiple times, the performance metrics reported in experiments correctly represent validation accuracy rather than the model's true test set accuracy.

Validation Accuracy vs. True Test Set Accuracy

Validation error is a metric used to evaluate a model's performance on a separate holdout dataset throughout the training process. By checking the model's accuracy on unseen data, typically after each epoch, practitioners can determine if the model is generalizing well or if it is starting to overfit the training data. This metric is centrally used to define the patience criterion for early stopping.

Definition of Validation Error

A validation dataset (or validation set) is a separate partition of data used to assess the quality of a machine learning model's solution. After training a model iteratively on the training data, its predictive performance is evaluated on this distinct validation dataset to measure how well it has learned the underlying patterns. This evaluation is critical for automatically tuning hyperparameters without relying on the training set, ensuring the model generalizes appropriately to unseen data.

University of Michigan - Ann Arbor

Claude

1. Validation set
2. Leave-One-Out Cross-Validation
3. k-fold Cross-Validation

Cross-Validation resampling methods

- Training set: to train the model
- Validation set (development set): to tune the model parameters
- Test set: to test the performance of the model on unobserved data

Division of dataset in supervised statistical learning

In machine learning, hyperparameters are user-defined, tunable settings that govern the structure or training process of a model but are not updated within the primary training loop itself. Examples include the number of epochs (`max_epochs`), the minibatch size (`batch_size`), and the learning rate (`lr`). Because these values are not optimized by the standard training algorithm along with the model's weights and biases, they must be specified beforehand. Despite not being learned during training, they still significantly influence the performance of the model, affecting both optimization during training and generalization to unseen data.

Hyperparameter

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Dive into Deep Learning

If we have $n$ data points in our training set, we train the model on $n-1$ data points. Then, we make a prediction on the observation that we left out. We repeat this process repeats $n$ times, and we will average the performance of our model for each of the predictions that we make. This average will give us model performance.

Leave-One-Out Cross-Validation

To perform K-fold CV on a dataset with $$n$$ observations, the training set is separated into $$k$$ sets by randomly sampling $$n/k$$ observations from the dataset into each fold. 
The model of interest is then trained on $$k−1$$ folds and validated on the left-out fold, i.e., once the model is trained, we use it to make predictions on the group that was left out of training. This process repeats $n$ times. To get the model performance, we average the results from all of the tests. K-Fold CV proves a good measure of the empirical testing error of a model. 


K-Fold Cross-Validation

The advantage of k-fold CV is that it often gives more accurate estimates of the test error rate than does LOOCV. This has to do with a bias-variance trade-off.
The validation set approach can lead to overestimates of the test error rate, since in this approach the training set used to fit the statistical learning method contains only half the observations of the entire data set ( it means the model does not have enough information about the data set.)
LOOCV will give approximately unbiased estimates of the test error since each training set contains n − 1 observation, which is almost as many as the number of observations in the full data set.
K-fold CV for, say, k = 5 or k = 10 will lead to an intermediate level of
bias, since each training set contains (k − 1)n/k observations—fewer than
in the LOOCV approach, but substantially more than in the validation set
approach.

Note:
LOOCV has a higher variance than does K-fold CV with k < n.

Bias-Variance Trade Off Comparison

Stratified folds each contain a proportion of classes that matches the overal dataset. Now, all cclasses will be fairly represented in the test set.

Stratified Cross-Validation

Which of the following is true of cross-validation?

Validation Dataset

The observations that are used in a method to build a model for estimating the response variable.

Training data

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make an assumptions that the examples in each dataset are independent from each other, and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example. 

The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{data}$. 

This probabilistic framework and the assumption enable us to mathematically study the relationship between training error and test error.

Data-Generating Process and Data-Generating Distribution (in Machine Learning)

This function randomly shuffles the dataset and splits off a certain percentage of the input samples for use as a training set, and then puts the remaining samples into a different variable for use as a test set. So in this example, we're using a 75-25% split of training versus test data. And that's a pretty standard relative split that's used. It's a good rule of thumb to use in deciding what proportion of training versus test might be helpful.
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
- Unless optional parameters test_size or train_size are passed in, the method will subset the training and test data randomly into 75% train:25% test
- random_state is an optional parameter used for reproducible output across multiple function calls

Train Test Split Function

- Data is divided 80% as training set, 10% as development set, and 10% as test set.
- Test set should be as large as possible, since a small test set may be accidentally unrepresentative.
- A good common practice is to find the smallest test set that gives us enough statistical power to measure a significant difference between models, since we also need the training set to include as much as data.

Common Practices of Train/Test Set Arrangements in NLP

A test dataset is a separate partition of data that is held out from the learning process and used strictly to evaluate a model's performance. Because performing well on the training data does not guarantee success on unseen data, the test set serves as a critical measure of the model's true ability to generalize.

Learn Before

Related

Learn After