1. Let $$M_0$$ denote the null model, which contains no predictors.
2. For k = 0, 1, ..., p-1:
(a) Consider all p-k models that augment the predictors in $$M_k$$ with one additional predictor.
(b) Choose the best among these p-k models, and call it $$M_{k+1}$$. Here best is defined as having smallest RSS or highest $$R^2$$.
3. Select a single best model from among $$M_0, ..., M_p$$ using cross-validated prediction error, $$C_p$$ (AIC), BIC, or adjusted $$R^2$$.

Forward Stepwise Selection Algorithm

Forward stepwise selection begins with a model with no inputs and adds the input that improves the model the greatest one-by-one until all the necessary nodes are in the model.

University of Michigan - Ann Arbor

More often than not, there are predictors in our data set that are not associated with the response. Because of this it is useful to narrow down our model to include only a subset of predictors.
- Forward Selection
- Backward Selection
- Mixed Selection


Methods for deciding important variable

In most basic terms, subset selection has two steps:
1. Identify a subset of predictors related to response
2. Use least squares on our reduced set of variables to fit a new model

Subset Selection

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Similar to forward stepwise selection, the hybrid technique will begin with no inputs and add the most beneficial inputs one-by-one. However, after each input addition, the hybrid method may remove inputs that no longer improve the model. 

Hybrid Stepwise Selection

Backwards stepwise selection begins with a model with all inputs included and removes the least beneficial input one-by-one.

Backwards Stepwise Selection

Forward Stepwise Selection

1. Best subset selection
2. Stepwise model selection 

Types of Subset Selection

If we have p variables, the number of models we need to fit for all subset selection methods:
1. Best Subset Selection: $$2^p$$
2. Forward/Backward Stepwise Selection: $$1+p(p+1)/2$$

For example, when p = 20, best subset selection requires fitting 1,048,475 models, whereas forward stepwise selection requires fitting only 211 models.

Learn Before

Related

Learn After