If we have p variables, the number of models we need to fit for all subset selection methods:
1. Best Subset Selection: $$2^p$$
2. Forward/Backward Stepwise Selection: $$1+p(p+1)/2$$

For example, when p = 20, best subset selection requires fitting 1,048,475 models, whereas forward stepwise selection requires fitting only 211 models.

University of Michigan - Ann Arbor

In most basic terms, subset selection has two steps:
1. Identify a subset of predictors related to response
2. Use least squares on our reduced set of variables to fit a new model

Subset Selection

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

1. Best subset selection
2. Stepwise model selection 

Types of Subset Selection

Forward stepwise selection begins with a model with no inputs and adds the input that improves the model the greatest one-by-one until all the necessary nodes are in the model.

Forward Stepwise Selection

Similar to forward stepwise selection, the hybrid technique will begin with no inputs and add the most beneficial inputs one-by-one. However, after each input addition, the hybrid method may remove inputs that no longer improve the model. 

Hybrid Stepwise Selection

Backwards stepwise selection begins with a model with all inputs included and removes the least beneficial input one-by-one.

Learn Before

Related