In statistical learning, a consistent predictor is an algorithm or model that, under mild conditions, will eventually converge to the optimal predictor as the size of the training data increases to infinity. This theoretical guarantee means that the algorithm, such as the $$1$$-nearest neighbor method, will ultimately find the best possible mapping from inputs to outputs when provided with sufficient data.

Consistent Predictor

A model with the goal of making accurate predictions based on input data. Can use PSIS, WAIC, and CV Benchmarks to determine predictive accuracy. These models tend to have waek bacdoor paths that provide little to no information about cause.

Arizona State University

Need to identify goal:
1.  Making Predictions
2. Inferring Cause

Model Selection

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

Does a good PSIS or WAIC score indicate a good causal model?

Causal Model

Predictive Model

Many times identifying and addressing outliers in model selection requires a one-by-one approach. While dropping outliers is a commonly used strategy, one must make sure they fit the model first. This means you must avoid dropping data points purely based on standard deviations. Sometimes multiple outliers are needed to properly fit data, so you should always fit the model first then make necessary drops.

Effects of Outliers on Model selection

In the field of machine learning, benchmark datasets are frequently reused over long periods of time to evaluate and compare different algorithms, such as in image classification and optical character recognition. Because researchers can rarely afford to gather a completely new test set for every experiment, this practice of data recycling effectively turns the test data into validation data, blurring the strict theoretical boundaries between the two.

Recycling Benchmark Data

The methodological flaw known as training on the test set occurs when a test dataset is repeatedly used to evaluate and select models, inadvertently causing the model to overfit to the test data. To mitigate this issue, standard practice requires splitting the original data into three separate partitions by introducing a distinct validation set, which is used exclusively for hyperparameter tuning while keeping the true test set completely unseen.

Learn Before

Related

Learn After