Overfitting occurs when a model performs significantly better on its training data than on validation or holdout data, resulting in a large generalization gap. This indicates that the model is overly complex and has begun to memorize noise rather than learning generalizable patterns. However, severe overfitting is not always inherently detrimental in deep learning; the best predictive models often exhibit this behavior. The primary goal is to drive the overall generalization error lower, meaning the gap is only problematic if it obstructs that goal.

Overfitting

- There are few parameters to work with the model. 
- Poor variance explanation or low R-squared
- Variance/Bias trade results in high bias-low variance. 

Underfitting 

- These concepts are related to model fitting within or outside Bayesian Statistics

- Direct relation to the number of parameters used to build a model

- There is a preference to use simpler models (Ockham’s razor), then we should ask when is too simple?

- There is a measure for model fitting, R-squared, which is not always an indicator of a good fit.


University of Rochester

- Statistical Golems
- Plausibility
- Posterior Distribution
- Sampling the Imaginary
- Building a Bayesian Model
- Gaussian Distribution
- Correlation vs. Causation
- Linear Models
- Multiple Regression Models
- Perils of Multiple Regression
- Underfitting vs Overfitting 
- Entropy and Accuracy
- Interactions
- Markov Chain Monte Carlo
- Generalized Linear Models



Concepts in Bayesian Statistics

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

- Scientific models are designed for making calculations, but they cannot determine causality on their own. 
- To determine the most appropriate statistical model to use, researchers ask questions about the type of data they have and what they wish to measure. 
- Frequentist approach is based on re-sampling methods, and it is a special case of Bayesian analysis
- We have models that are defined to do certain tests, but as researchers, you need to get behind the methods and determine if results are practically significant 

Statistical Golems 

- Bayesian Statistics considers all possible paths that yield different outcomes
- Plausibilities are probabilities – prior plausibility and posterior plausibility 

Plausibility

The posterior distribution is a combination of data, likelihood, parameters, and prior. It takes on the form of a conditional probability. 

Posterior Distribution

- Bayesian inference uses probabilities, and through Bayesian updating, one can obtain posterior plausibilities  

Building a Bayesian Model

Correlation is a relationship, yet causality indicates one event is the result of the occurence of another event.

Correlation is not equal to causality

- Comprises the likelihood in simple linear regression models
- Counts up the relative numbers of ways different combinations of means and standard deviations can produce an observation

Gaussian Distribution

- Predictor variable
- Linear model strategy
- Finding the prior 
- Finding the posterior distribution 
- Generating predictions and intervals from posterior of a fit model 

Linear Predictions

Below are some of the potential issues when adding variables to a regression model without a clear idea of the causal model - 

- Multicollinearity
- Post-Treatment Bias
- Collider Bias
- How  to recognize and prevent these confounding issues

Perils of Multiple Regression

- Sampling from a grid-approximate posterior
- Sampling to summarize
- Sampling to simulate prediction

Sampling the Imaginary

Underfitting vs Overfitting

Accuracy: 

- Heuristic approach to the concept, we can have a high probability of making correct predictions and predict wrong most of the time yielding poor accuracy. 

-  Log scoring Rule for accuracy measure inherently uses the Bayes Theorem with joint probabilities (likelihood). 

Entropy: 

-  The uncertainty contained in a probability distribution. 

- Average log-prob of the event

-  Is closely related to its scientific meaning, the measure of disorder in a system, now in terms of probability. 



Entropy and Accuracy 

Thinking about how the symmetry is broken can be productive. It would help us to compare the interaction between two choices when we miss other information.

Symmetry of Interactions

If interaction exists, it is not enough to interpret the model with only posterior means and standard deviations. Thus, we need some methods to construct and interpret continuous interactions.

Continuous Interactions

- A class of techniques or methods that use a stochastic process to estimate a posterior probability – often used to generate samples/ observations from a posterior 
- STAN – Most common tool to run MCMC algorithms 
- Rstan and Rethinking package – can be used for simple models

Common MCMC Algorithms - 
- Metropolis/ Metropolis Hastings
- Gibbs Sampling 
- Hamiltonian Monte Carlo


Markov Chain Monte Carlo

Given some constraints on the parameters, the least informative distribution consistent with the constraints on the outcome variables will be nominated, according to the maximum entropy principle.

Maximum Entropy Priors 

GLM offers additional techniques for different types of data such as:
- Continuous (OLS)
- counts 
- dichotomous (two-categories) 

GLM Components:

- Link Function - serves as a model regulator
- Linear Predictor - Data fitting 
- Random Error

Exponential Family

- Every member of this family is a maximum entropy distribution,  for some set of constraints
- GLM Structure is the same for each member. 


GLM and Exponential Family 

Goal: Map linear model to [0,1];
Log-odds is natural parameter;


Rethink: Logit Link

Multiple regression is a statistical technique that extends simple regression by measuring several predictor variables ($$X_1, X_2, X_3, \dots, X_i$$) and using them to predict a single outcome variable ($$Y$$). It can also be utilized to describe the complex relationship between the outcome variable and a set of predictors. The result of this analysis is an equation that expresses the outcome variable as an additive combination of all the measured predictor variables.

Multiple Regression

In experimental research, an interaction effect (or simply an interaction) exists when the influence of one independent variable on a measured outcome is fundamentally altered depending on the specific level of a second independent variable. This indicates that the variables do not operate in isolation; rather, they combine to produce complex effects that cannot be fully explained by analyzing them independently.

Learn Before

Related

Learn After