The methodological flaw known as training on the test set occurs when a test dataset is repeatedly used to evaluate and select models, inadvertently causing the model to overfit to the test data. To mitigate this issue, standard practice requires splitting the original data into three separate partitions by introducing a distinct validation set, which is used exclusively for hyperparameter tuning while keeping the true test set completely unseen.

Claude

Need to identify goal:
1.  Making Predictions
2. Inferring Cause

Model Selection

Dive into Deep Learning

Does a good PSIS or WAIC score indicate a good causal model?

Causal Model

A model with the goal of making accurate predictions based on input data. Can use PSIS, WAIC, and CV Benchmarks to determine predictive accuracy. These models tend to have waek bacdoor paths that provide little to no information about cause.

Predictive Model

Many times identifying and addressing outliers in model selection requires a one-by-one approach. While dropping outliers is a commonly used strategy, one must make sure they fit the model first. This means you must avoid dropping data points purely based on standard deviations. Sometimes multiple outliers are needed to properly fit data, so you should always fit the model first then make necessary drops.

Effects of Outliers on Model selection

In the field of machine learning, benchmark datasets are frequently reused over long periods of time to evaluate and compare different algorithms, such as in image classification and optical character recognition. Because researchers can rarely afford to gather a completely new test set for every experiment, this practice of data recycling effectively turns the test data into validation data, blurring the strict theoretical boundaries between the two.

Learn Before

Related