Does a good PSIS or WAIC score indicate a good causal model?

Causal Model

A model with the goal of making accurate predictions based on input data. Can use PSIS, WAIC, and CV Benchmarks to determine predictive accuracy. These models tend to have waek bacdoor paths that provide little to no information about cause.

Predictive Model

Many times identifying and addressing outliers in model selection requires a one-by-one approach. While dropping outliers is a commonly used strategy, one must make sure they fit the model first. This means you must avoid dropping data points purely based on standard deviations. Sometimes multiple outliers are needed to properly fit data, so you should always fit the model first then make necessary drops.

Effects of Outliers on Model selection

In the field of machine learning, benchmark datasets are frequently reused over long periods of time to evaluate and compare different algorithms, such as in image classification and optical character recognition. Because researchers can rarely afford to gather a completely new test set for every experiment, this practice of data recycling effectively turns the test data into validation data, blurring the strict theoretical boundaries between the two.

Recycling Benchmark Data

The methodological flaw known as training on the test set occurs when a test dataset is repeatedly used to evaluate and select models, inadvertently causing the model to overfit to the test data. To mitigate this issue, standard practice requires splitting the original data into three separate partitions by introducing a distinct validation set, which is used exclusively for hyperparameter tuning while keeping the true test set completely unseen.

Training on the Test Set

Need to identify goal:
1.  Making Predictions
2. Inferring Cause

Claude

Arizona State University

- Compare models through cross validation and information criteria to determine the predictive accuracy and tendency for overfitting, in addition to highlighting any observations that are exerting significant influence to the model 
- Overfitting: simple models may lead to more accurate predictions than more complicated models

Model comparison and prediction

Dive into Deep Learning

A "skeptical" prior that, when tuned properly,  slows the rate of learning from the sample. This reduces overfitting while allowing the model to learn the regular features of the sample.



Regularization Priors

Two strategies:
- Cross-Validation
- Information Criteria

These stategies guess how well model will predict data.

Learn Before

Related

Learn After