- Uses powers of a variable (squares and cubes) as extra predictors 
- Need to first standardize the predictor variable 
Considerations: 
- A better fit to the sample does not necessarily mean a better model
- Fitting a causal relationship gives us no new information about the model 

Polynomial regression

- Spline: a smooth function built out of smaller, component functions 
- B-splines divide the range of a predictor variable into parts
- Assign a parameter to each part and the parameters are turned on and off so they sum to a fancy curve 
- Create new predictor variables that make up a basis function

Basis-Splines (B-Splines)

Multiple regression model looks like the polynomial model which was introduced before but we will dive into it, and use this method to figure out "correlation". 
Multiple regression have more independent varialbles, or explanatory variables, to determine the output (the dependent variable)

Multiple Regression Model 

The multiple regression equation expresses the outcome variable as an additive combination of several predictor variables. Its general form is: $$Y = b_1X_1 + b_2X_2 + b_3X_3 + \dots + b_iX_i$$, where $$Y$$ is the predicted outcome, the $$X$$ terms are the individual predictor variables, and the corresponding $$b$$ terms are their regression weights.

Multiple Regression Formula

The primary advantage of multiple regression is its ability to show whether a specific predictor variable makes a unique contribution to an outcome variable over and above the contributions made by other predictors. This allows researchers to statistically control for other variables while examining a specific relationship.

Advantage of Multiple Regression

If a researcher investigates how income and health relate to happiness, they face a confound because income and health are correlated. By using a multiple regression analysis that includes both as predictor variables, the researcher can statistically control for one while examining the other. This determines whether the part of income unrelated to health predicts happiness, and whether the part of health unrelated to income predicts happiness.

Example of Multiple Regression: Income, Health, and Happiness

Which of the following best describes the primary function of a multiple regression analysis?

Multiple regression is a statistical technique that extends simple regression by measuring several predictor variables ($$X_1, X_2, X_3, \dots, X_i$$) and using them to predict a single outcome variable ($$Y$$). It can also be utilized to describe the complex relationship between the outcome variable and a set of predictors. The result of this analysis is an equation that expresses the outcome variable as an additive combination of all the measured predictor variables.

University of Michigan - Ann Arbor

Google

- Statistical Golems
- Plausibility
- Posterior Distribution
- Sampling the Imaginary
- Building a Bayesian Model
- Gaussian Distribution
- Correlation vs. Causation
- Linear Models
- Multiple Regression Models
- Perils of Multiple Regression
- Underfitting vs Overfitting 
- Entropy and Accuracy
- Interactions
- Markov Chain Monte Carlo
- Generalized Linear Models



Concepts in Bayesian Statistics

Statistical regression is a technique that allows researchers to predict the value of one variable given the value of another variable. While correlation coefficients describe the strength and direction of a relationship, regression utilizes that established relationship to make specific mathematical predictions.

Statistical Regression

In statistical regression, the predictor variable is the variable that is used to make a prediction about another variable. It serves as the known input value from which the predicted outcome is derived.

Predictor Variable

In regression analysis, the outcome variable—also referred to as the criterion variable—is the variable that is being predicted by the model.

Outcome Variable or Criterion Variable

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd edition)

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

- Scientific models are designed for making calculations, but they cannot determine causality on their own. 
- To determine the most appropriate statistical model to use, researchers ask questions about the type of data they have and what they wish to measure. 
- Frequentist approach is based on re-sampling methods, and it is a special case of Bayesian analysis
- We have models that are defined to do certain tests, but as researchers, you need to get behind the methods and determine if results are practically significant 

Statistical Golems 

- Bayesian Statistics considers all possible paths that yield different outcomes
- Plausibilities are probabilities – prior plausibility and posterior plausibility 

Plausibility

The posterior distribution is a combination of data, likelihood, parameters, and prior. It takes on the form of a conditional probability. 

Posterior Distribution

- Bayesian inference uses probabilities, and through Bayesian updating, one can obtain posterior plausibilities  

Building a Bayesian Model

Correlation is a relationship, yet causality indicates one event is the result of the occurence of another event.

Correlation is not equal to causality

- Comprises the likelihood in simple linear regression models
- Counts up the relative numbers of ways different combinations of means and standard deviations can produce an observation

Gaussian Distribution

- Predictor variable
- Linear model strategy
- Finding the prior 
- Finding the posterior distribution 
- Generating predictions and intervals from posterior of a fit model 

Linear Predictions

Below are some of the potential issues when adding variables to a regression model without a clear idea of the causal model - 

- Multicollinearity
- Post-Treatment Bias
- Collider Bias
- How  to recognize and prevent these confounding issues

Perils of Multiple Regression

- Sampling from a grid-approximate posterior
- Sampling to summarize
- Sampling to simulate prediction

Sampling the Imaginary

- These concepts are related to model fitting within or outside Bayesian Statistics

- Direct relation to the number of parameters used to build a model

- There is a preference to use simpler models (Ockham’s razor), then we should ask when is too simple?

- There is a measure for model fitting, R-squared, which is not always an indicator of a good fit.


Underfitting vs Overfitting

Accuracy: 

- Heuristic approach to the concept, we can have a high probability of making correct predictions and predict wrong most of the time yielding poor accuracy. 

-  Log scoring Rule for accuracy measure inherently uses the Bayes Theorem with joint probabilities (likelihood). 

Entropy: 

-  The uncertainty contained in a probability distribution. 

- Average log-prob of the event

-  Is closely related to its scientific meaning, the measure of disorder in a system, now in terms of probability. 



Entropy and Accuracy 

Thinking about how the symmetry is broken can be productive. It would help us to compare the interaction between two choices when we miss other information.

Symmetry of Interactions

If interaction exists, it is not enough to interpret the model with only posterior means and standard deviations. Thus, we need some methods to construct and interpret continuous interactions.

Continuous Interactions

- A class of techniques or methods that use a stochastic process to estimate a posterior probability – often used to generate samples/ observations from a posterior 
- STAN – Most common tool to run MCMC algorithms 
- Rstan and Rethinking package – can be used for simple models

Common MCMC Algorithms - 
- Metropolis/ Metropolis Hastings
- Gibbs Sampling 
- Hamiltonian Monte Carlo


Markov Chain Monte Carlo

Given some constraints on the parameters, the least informative distribution consistent with the constraints on the outcome variables will be nominated, according to the maximum entropy principle.

Maximum Entropy Priors 

GLM offers additional techniques for different types of data such as:
- Continuous (OLS)
- counts 
- dichotomous (two-categories) 

GLM Components:

- Link Function - serves as a model regulator
- Linear Predictor - Data fitting 
- Random Error

Exponential Family

- Every member of this family is a maximum entropy distribution,  for some set of constraints
- GLM Structure is the same for each member. 


GLM and Exponential Family 

Goal: Map linear model to [0,1];
Log-odds is natural parameter;


Rethink: Logit Link

Multiple Regression

In experimental research, an interaction effect (or simply an interaction) exists when the influence of one independent variable on a measured outcome is fundamentally altered depending on the specific level of a second independent variable. This indicates that the variables do not operate in isolation; rather, they combine to produce complex effects that cannot be fully explained by analyzing them independently.

Interaction Effect

In psychological research, what is the primary function of statistical regression?

While a correlation coefficient describes the strength and direction of a relationship, researchers use statistical ____ to make specific mathematical predictions about the value of one variable given another.

In a statistical regression analysis, what is the specific role of a predictor variable?

Simple regression is a statistical technique that involves using exactly one predictor variable to predict the value of a single outcome variable.

Learn Before

Related

Learn After