In most basic terms, subset selection has two steps:
1. Identify a subset of predictors related to response
2. Use least squares on our reduced set of variables to fit a new model

Subset Selection

Process of reducing the number of random variables under consideration by obtaining a set of principal variables. Dimension reduction methods transform the variables before fitting a least squares model. The two types of dimension reduction are
1. Feature selection, 
which selects a subset of variables from the original dataset, making regression or classification more accurate.
2. Feature extraction, 
which combines variables into features, reducing the amount of data processed while maintaining the accuracy of the original dataset. 

Dimension Reduction

$R_{adj}^2 = 1-(1-R^2)*(n-1)/(n-p-1)$   

Adjusted R-Squared is a version of R-Squared that adjusted for the number of predictors (independent variables) in a model. As we know, $R^2$ always increase if we adding more predictors. This Adjusted R-Squared has an advantage over the normal R-Squared metric because it accounts for statistical shrinkage and the normal R-Squared meter tends to hurt more when more independent variables occur in the system. This adjusted R-square allows us to do predictors selection and can be a metrics for forward/backward selection.


Adjusted R-Square Explained 

AIC = $ \frac{RSS+2d \hat{\sigma}}{n \hat{\sigma ^2}}$

The AIC (Akaike Information Criterion) Metric describes the quality of the model with the data that is given. AIC is applicable in a board range of modeling frameworks as it only requires large sample properties of maximum likelihood estimator. It uses candidate models to manipulate the data but does not require the assumption that these models be true or correct. AIC is the trade-off between goodness of fit and complexity of the variables that are considered in the problem. R-Squared changes relative to the complexity of the system (variables) but AIC does not. 

Note: AIC cannot be used to compare models with different dataset. For example, if one model involves transformations on response variable (like log transformations) and another one does not, one cannot use AIC to compare two models. 

AIC Explained

BIC = $$\frac{RSS+log(n)d \hat{\sigma^2}}{n \hat{\sigma^2}}$$

Bayesian information criterion (BIC) is a criterion for model selection among a finite set of models. The models can be tested using corresponding BIC values. Lower BIC value indicates lower penalty terms hence a better model. It is worth to know BIC closely relates to AIC. The only difference is BIC considers the number of observations in the formula, which AIC does not.

More details: https://www.immagic.com/eLibrary/ARCHIVES/GENERAL/WIKIPEDI/W120607B.pdf

Bayesian Information Criterion

Methods:
- Subset selection
- Shrinkage
- Dimension reduction

Criteria:
(a) Indirectly estimate test error by making adjustment to the training error:
- Adjusted R-Squared
- AIC
- BIC
- Mallow’s $C_p$
(b) Directly estimate the test error:
- Cross-validation

University of Michigan - Ann Arbor

1. Is the relation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ causal?
2. Is the correlation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ linear or non-linear?
3. Which predictors $X_1, X_2, X_3, ..., X_P$ are correlated, and which ones are not correlated with outcome $Y$?
4. What is the direction of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?
5. What is the magnitude of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?

Types of inferential questions in statistical learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Methods of choosing important predictors to improve interpretability

Depending on the values of other predictors in a multiple linear regression model, the coefficient of a predictor may be positive, indicating an association between an increase in the value of this predictor and an increase in the value of $Y$, or the coefficient can may be negative, indicating an association between a decrease in the value of this predictor and a decrease in the value of $Y$.

Direction of correlation between each predictor and outcome

Causal inference refers to making conclusions about finding the reasonings behind the relationship of two or more variables. After coming to some conclusions about statistical data, we ask why such relationships exist between variables.

Learn Before

Related

Learn After