Which dataset best measures how well a model handles new examples from the same distribution as its training data?

Training error is computed by evaluating the model on the same examples used to fit it.

Which Set Measures the Performance You Care About?

Match each dataset in the four-way evaluation setup to what it is mainly used to measure.

Order the evaluations used to move from training fit to the final assessment of performance on the target distribution.

Which dataset or datasets should be used to estimate how well a model will perform on the distribution you ultimately care about?

A training-dev set should come from the same distribution as the dev and test sets so it measures performance in the target environment.

Training-Distribution Development Set

Connect Each Dataset to Its Evaluation Purpose

Using the Four-Dataset Framework to Isolate the Main Weakness

How do the different datasets in a four-set evaluation scheme serve different diagnostic purposes?

What each split shows in a four-way classifier check

What the training dev set tells you