1. Let $$M_p$$ denote the null model, which contains no predictors.
2. For k = p, p-1, ..., 1:
(a) Consider all k models that contain all but one of the predictors in $$M_k$$ for a total of k - 1 predictors.
(b) Choose the best among these k models, and call it $$M_{k-1}$$. Here best is defined as having smallest RSS or highest $$R^2$$.
3. Select a single best model from among $$M_0, ..., M_p$$ using cross-validated prediction error, $$C_p$$ (AIC), BIC, or adjusted $$R^2$$.

Restriction: requires that the number of samples n is larger than the number of variables p so that the full model can be fit.

Backward Stepwise Selection Algorithm

Backwards stepwise selection begins with a model with all inputs included and removes the least beneficial input one-by-one.

University of Michigan - Ann Arbor

More often than not, there are predictors in our data set that are not associated with the response. Because of this it is useful to narrow down our model to include only a subset of predictors.
- Forward Selection
- Backward Selection
- Mixed Selection


Methods for deciding important variable

In most basic terms, subset selection has two steps:
1. Identify a subset of predictors related to response
2. Use least squares on our reduced set of variables to fit a new model

Subset Selection

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Similar to forward stepwise selection, the hybrid technique will begin with no inputs and add the most beneficial inputs one-by-one. However, after each input addition, the hybrid method may remove inputs that no longer improve the model. 

Hybrid Stepwise Selection

Backwards Stepwise Selection

Forward stepwise selection begins with a model with no inputs and adds the input that improves the model the greatest one-by-one until all the necessary nodes are in the model.

Forward Stepwise Selection

1. Best subset selection
2. Stepwise model selection 

Types of Subset Selection

If we have p variables, the number of models we need to fit for all subset selection methods:
1. Best Subset Selection: $$2^p$$
2. Forward/Backward Stepwise Selection: $$1+p(p+1)/2$$

For example, when p = 20, best subset selection requires fitting 1,048,475 models, whereas forward stepwise selection requires fitting only 211 models.

Learn Before

Related

Learn After