Method of fitting regression models in which the choice of predictive variables is carried out by an automatic procedure - adding or subtracting variables at each step depending on the results of predetermined tests 
Examples of tests that could be used:
1. F-tests
2. t-tests
3. R Squared

Stepwise Model Selection

Approach that includes testing all possible combinations of predictor variables and then selecting the best model.
Required packages for R:
1. tidyverse for easy data manipulation and visualization
2. caret for easy machine learning workflow
3. leaps, for computing best subsets regression

Best Subset Selection

1. Best subset selection
2. Stepwise model selection 

University of Michigan - Ann Arbor

In most basic terms, subset selection has two steps:
1. Identify a subset of predictors related to response
2. Use least squares on our reduced set of variables to fit a new model

Subset Selection

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Types of Subset Selection

Forward stepwise selection begins with a model with no inputs and adds the input that improves the model the greatest one-by-one until all the necessary nodes are in the model.

Forward Stepwise Selection

Similar to forward stepwise selection, the hybrid technique will begin with no inputs and add the most beneficial inputs one-by-one. However, after each input addition, the hybrid method may remove inputs that no longer improve the model. 

Hybrid Stepwise Selection

Backwards stepwise selection begins with a model with all inputs included and removes the least beneficial input one-by-one.

Backwards Stepwise Selection

If we have p variables, the number of models we need to fit for all subset selection methods:
1. Best Subset Selection: $$2^p$$
2. Forward/Backward Stepwise Selection: $$1+p(p+1)/2$$

For example, when p = 20, best subset selection requires fitting 1,048,475 models, whereas forward stepwise selection requires fitting only 211 models.

Learn Before

Related

Learn After