Methods:
- Subset selection
- Shrinkage
- Dimension reduction

Criteria:
(a) Indirectly estimate test error by making adjustment to the training error:
- Adjusted R-Squared
- AIC
- BIC
- Mallow’s $C_p$
(b) Directly estimate the test error:
- Cross-validation

Methods of choosing important predictors to improve interpretability

Depending on the values of other predictors in a multiple linear regression model, the coefficient of a predictor may be positive, indicating an association between an increase in the value of this predictor and an increase in the value of $Y$, or the coefficient can may be negative, indicating an association between a decrease in the value of this predictor and a decrease in the value of $Y$.

Direction of correlation between each predictor and outcome

Causal inference refers to making conclusions about finding the reasonings behind the relationship of two or more variables. After coming to some conclusions about statistical data, we ask why such relationships exist between variables.

Causal Inference

1. Is the relation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ causal?
2. Is the correlation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ linear or non-linear?
3. Which predictors $X_1, X_2, X_3, ..., X_P$ are correlated, and which ones are not correlated with outcome $Y$?
4. What is the direction of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?
5. What is the magnitude of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?

University of Michigan - Ann Arbor

Inference means understanding the association of changes in the outcome $Y$ and changes in the predictors matrix $X$, i.e., estimating the characteristics of a function $f$ that maps the values of $X$ to corresponding values of $Y$.

Inference in statistical learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

In both prediction and inference, we estimate a function $f$ that maps the values of predictors $X$ to outcome $Y$.
In prediction, our goal is to estimate the function $f$ based on the available values of $X$ and corresponding $Y$ so that we can use $f$ to predict the values of $Y$ for the values of $X$ for which we don't know the corresponding values of $Y$. So, the function $\hat{f}$ can be treated as a black box, and our main objectives are to improve the accuracy and precision of the prediction.
In inference, we already know all the corresponding values of $X$ and $Y$, and we're interested in understanding the characteristics of the function $f$ that maps the values of $X$ to their corresponding values of $Y$. So, $\hat{f}$ cannot be treated as a black box and our main objective is to find the exact functional form of $\hat{f}$.

Inference vs. prediction in statistical learning

Types of inferential questions in statistical learning

The inference engine is a machine that accepts Assumptions, Queries, and Data, and produces Testable implications, Estimand, and Estimate. The implications and estimand do not necessarily need to be presented at the end, but they are both implicit within the model. The first of the outputs is a Y/N decision as to whether the given query could be answered under the existing causal model, assuming perfect data. If the answer is Yes, the inference engine next produces an Estimand, a mathematical formula that generates the answer from any hypothetical data. If the answer is No, then the engine returns to the assumptions stage to refine the model, through either simplifying assumptions or adding new scientific knowledge. Finally, after the inference engine has received the Data, it will use the Estimand to produce an actual Estimate for the answer, along with the amount of uncertainty in that estimate.

Inference Engine

The calculus of variations is a mathematical tool that helps with solving an optimization problem with respect to a function. 

Calculus of variations

Exact inference can be described as an optimization problem. Thus, approximate inference algorithms can be derived by approximating the underlying optimization. We assume a probabilistic model consisting of observed variables $v$ and latent variables $h$. Sometimes it is too difficult to compute the log-probability ($log p(v;\theta)$) of the observed data, so we can instead compute a lower bound $\mathcal{L}(v, \theta,q)$ on $log p(v;\theta)$, called the evidence lower bound (ELBO), or the negative variational free energy. 

Inference as an Optimization Problem

Approximate Inference affects the learning process which affects the accuracy of the algorithm. A behavior is seen in which approximating assumptions become self-fulfilling. Training the model with a unimodal approximate posterior will result in a posterior that is closer than would result from training with exact inference. Computing the harm by variational approximation is very difficult.

Inference Effects on Learning

Linear models are a good choice because they allow us to easily understand the relationship between X1, X2, X3.... and Y. These types of linear models are more restrictive and therefore easier to interpret. However, they do not allow us to consider statistical situations in which nonlinear relationships exist. 
Reference: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An introduction to statistical learning: with applications in R. New York: Springer.


Learn Before

Related

Learn After