Maximum entropy provides a way to generate a prior that embodies the background information.
Compared with Maximum Entropy Priors, GLM may not have informative parameters. Sometimes, the Maximum Entropy Principle allows us to have a better understanding of the world.

Application of Maximum Entropy Priors

Given some constraints on the parameters, the least informative distribution consistent with the constraints on the outcome variables will be nominated, according to the maximum entropy principle.

Ohio State University, Columbus

- Statistical Golems
- Plausibility
- Posterior Distribution
- Sampling the Imaginary
- Building a Bayesian Model
- Gaussian Distribution
- Correlation vs. Causation
- Linear Models
- Multiple Regression Models
- Perils of Multiple Regression
- Underfitting vs Overfitting 
- Entropy and Accuracy
- Interactions
- Markov Chain Monte Carlo
- Generalized Linear Models



Concepts in Bayesian Statistics

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

- Scientific models are designed for making calculations, but they cannot determine causality on their own. 
- To determine the most appropriate statistical model to use, researchers ask questions about the type of data they have and what they wish to measure. 
- Frequentist approach is based on re-sampling methods, and it is a special case of Bayesian analysis
- We have models that are defined to do certain tests, but as researchers, you need to get behind the methods and determine if results are practically significant 

Statistical Golems 

- Bayesian Statistics considers all possible paths that yield different outcomes
- Plausibilities are probabilities – prior plausibility and posterior plausibility 

Plausibility

The posterior distribution is a combination of data, likelihood, parameters, and prior. It takes on the form of a conditional probability. 

Posterior Distribution

- Bayesian inference uses probabilities, and through Bayesian updating, one can obtain posterior plausibilities  

Building a Bayesian Model

Correlation is a relationship, yet causality indicates one event is the result of the occurence of another event.

Correlation is not equal to causality

- Comprises the likelihood in simple linear regression models
- Counts up the relative numbers of ways different combinations of means and standard deviations can produce an observation

Gaussian Distribution

- Predictor variable
- Linear model strategy
- Finding the prior 
- Finding the posterior distribution 
- Generating predictions and intervals from posterior of a fit model 

Linear Predictions

Below are some of the potential issues when adding variables to a regression model without a clear idea of the causal model - 

- Multicollinearity
- Post-Treatment Bias
- Collider Bias
- How  to recognize and prevent these confounding issues

Perils of Multiple Regression

- Sampling from a grid-approximate posterior
- Sampling to summarize
- Sampling to simulate prediction

Sampling the Imaginary

- These concepts are related to model fitting within or outside Bayesian Statistics

- Direct relation to the number of parameters used to build a model

- There is a preference to use simpler models (Ockham’s razor), then we should ask when is too simple?

- There is a measure for model fitting, R-squared, which is not always an indicator of a good fit.


Underfitting vs Overfitting

Accuracy: 

- Heuristic approach to the concept, we can have a high probability of making correct predictions and predict wrong most of the time yielding poor accuracy. 

-  Log scoring Rule for accuracy measure inherently uses the Bayes Theorem with joint probabilities (likelihood). 

Entropy: 

-  The uncertainty contained in a probability distribution. 

- Average log-prob of the event

-  Is closely related to its scientific meaning, the measure of disorder in a system, now in terms of probability. 



Entropy and Accuracy 

Thinking about how the symmetry is broken can be productive. It would help us to compare the interaction between two choices when we miss other information.

Symmetry of Interactions

If interaction exists, it is not enough to interpret the model with only posterior means and standard deviations. Thus, we need some methods to construct and interpret continuous interactions.

Continuous Interactions

- A class of techniques or methods that use a stochastic process to estimate a posterior probability – often used to generate samples/ observations from a posterior 
- STAN – Most common tool to run MCMC algorithms 
- Rstan and Rethinking package – can be used for simple models

Common MCMC Algorithms - 
- Metropolis/ Metropolis Hastings
- Gibbs Sampling 
- Hamiltonian Monte Carlo


Markov Chain Monte Carlo

Maximum Entropy Priors 

GLM offers additional techniques for different types of data such as:
- Continuous (OLS)
- counts 
- dichotomous (two-categories) 

GLM Components:

- Link Function - serves as a model regulator
- Linear Predictor - Data fitting 
- Random Error

Exponential Family

- Every member of this family is a maximum entropy distribution,  for some set of constraints
- GLM Structure is the same for each member. 


GLM and Exponential Family 

Goal: Map linear model to [0,1];
Log-odds is natural parameter;


Rethink: Logit Link

Multiple regression is a statistical technique that extends simple regression by measuring several predictor variables ($$X_1, X_2, X_3, \dots, X_i$$) and using them to predict a single outcome variable ($$Y$$). It can also be utilized to describe the complex relationship between the outcome variable and a set of predictors. The result of this analysis is an equation that expresses the outcome variable as an additive combination of all the measured predictor variables.

Multiple Regression

In experimental research, an interaction effect (or simply an interaction) exists when the influence of one independent variable on a measured outcome is fundamentally altered depending on the specific level of a second independent variable. This indicates that the variables do not operate in isolation; rather, they combine to produce complex effects that cannot be fully explained by analyzing them independently.

Learn Before

Related

Learn After