Similar to forward stepwise selection, the hybrid technique will begin with no inputs and add the most beneficial inputs one-by-one. However, after each input addition, the hybrid method may remove inputs that no longer improve the model. 

Hybrid Stepwise Selection

Backwards stepwise selection begins with a model with all inputs included and removes the least beneficial input one-by-one.

Backwards Stepwise Selection

Forward stepwise selection begins with a model with no inputs and adds the input that improves the model the greatest one-by-one until all the necessary nodes are in the model.

Forward Stepwise Selection

More often than not, there are predictors in our data set that are not associated with the response. Because of this it is useful to narrow down our model to include only a subset of predictors.
- Forward Selection
- Backward Selection
- Mixed Selection


University of Michigan - Ann Arbor

Multiple Linear Regression is a linear regression that uses **two or more independent variables** to predict the outcome of a response variable. It is more effective and practical to predict or estimate dependent variables by the optimal combination of multiple independent variables than only one independent variable. 
$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + ... + \beta_p x_{ip} + \epsilon$$

Multiple Linear Regression

When dealing with data, whether analyzing, predicting, or anything else, the more is not always better.  Adding additional information leads to a higher dimension, meaning that the data points become more sparse.  The increased sparsity would make it harder to generalize accurately, which may lead to the "curse of dimensionality."


Curse of Dimensionality

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Methods for deciding important variable

Here is an example of multiple linear regression using R:
```r
library(MASS) #  MASS package is a very large collection of data sets and functions
library(ISLR) # ISLR package includes the data sets associated with this book
lm.fit=lm(medv∼lstat+age,data=Boston)# medv as a response, lstat and age as two predictors
summary(lm.fit) # gives us p-values and standard errors for the coeﬃcients, as well as the R2 statistic and F-statistic for the model

#The syntax lm(y∼x1+x2+x3) is used to ﬁt a model with three predictors, #x1, x2, andx3. The summary() function now outputs the regression #coeﬃcients for all the predictors. 

#We can also make a regression using all predictors with the following code:
lm.fit=lm(medv∼.,data=Boston) 
```

Example of Multiple Linear Regression Using R

```R 
#confint() provides information on confidence intervals for the coefficient estimates
#This can be used with both simple and multiple linear regression

confint(lm.fit)
```
Below is an example of the output information you can get by calling confint(lm.fit):

(from Pg. 111 of Introduction to Statistical Learning with Applications in R).

Learn Before

Related

Learn After