- PSIS and WAIC perform similarly when applied to ordinary linear models. Important differences are found in model types where the distribution is not approximately normal or in the presence of observations that strongly influence the posterior.

- CV and PSIS have higher variance as estimators of the KL divergence, while WAIC has greater bias. 

- PSIS has an advantage in warning the user when it is unreliable  

Comparing CV, PSIS, and WAIC

It provides an approximation of the out-of-sample deviance that converges to the cross-validation approximation ina large sample

Arizona State University

This approach uses information criteria to calculate an expected score out-of-sample. It constructs a theoretical estimate of the relative out-of-sample KL divergence. 

The best-known information criterion is the Akaike Information Criterion (AIC). This tells us is the dimensionality of the posterior distribution is a natural measure of the model's overfitting tendency.

However,  AIC is an approximation only reliable when:
- The prior are flat or overwhelmed by the likelihood 
- The posterior distribution is approximately multivariate Gaussian 
- The sample size (N) is much greater than the number of parameters (k)


Learn Before

Related

Learn After