Unlike classification models, classification trees do not suffer from overfitting in the event that a large B value is used. However, this requires choosing "a value of B sufficiently large that the error has settled down" (ISLR p.317).

Overfitting in classification trees

Bagging (bootstrap aggregation) can be applied to classification trees to improve prediction accuracy. While various approaches exist, the simplest involves recording the class predicted by each of the $$B$$ individual classification trees for a given test observation. The final prediction is determined by a majority vote, assigning the observation to the most commonly predicted class among the $$B$$ predictions.

University of Michigan - Ann Arbor

Google

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $$B$$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $$b$$th bootstrapped training data set to train the method to produce $$\hat{f}^{*b}(x)$$. Afterwards, all the predictions are then averaged in order to get $$\hat{f}_{bag}(x) = \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$$.

Bagging

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Bagging can be applied to decision trees, particularly with regression. It follows the same overall process: B regression trees are produced using B bootstrapped training sets. Afterwards, the predictions of all B regression trees are averaged in order to obtain a model that has reduced the variance of the B individual decision trees.

Bagging for Regression Trees

Bagging for Classification Trees

Out-of-Bag (OOB) error estimation is a technique used to evaluate the test error of a bagged model without requiring cross-validation. An observation is considered 'out-of-bag' if it was not used to fit a given bagged tree. On average, each observation is OOB for approximately $$B/3$$ of the $$B$$ total trees. To predict the response for the $$i$$th observation, the model uses only the subset of trees where that observation was OOB. For regression trees, these specific OOB predictions are averaged to yield a single prediction for the $$i$$th observation, whereas for classification trees, a majority vote is taken. Finally, these predictions are compared against the true responses to calculate the overall OOB error, such as the Mean Squared Error (MSE) for regression or the classification error rate.

Out-of-Bag Error Estimation

*Random forests* is quite similar to *bagging* with *decision trees* except that there is an improvement on spliting trees. They both use bootstrapped subset of training oberservations for a tree and repeated that process several times for the average/most common result. 

The difference lies in that, random forest only use randomly selected subsets of all predictors for spliting trees. This improvement *decorrelates* the trees and solves the problem of bagging that,  if there is a very strong predictor, most bagged trees might look similar and averaging could not reduce much variance for the highly correlated quantities. However, this can be addressed by decorrelating the trees thus reducing the variance and making the trees more reliable.

Random Forests vs. Bagging with Decision Trees

Boosting is heavily dependent on the trees formed before it, whereas in bagging, each tree is formed independently and varies instead based on the sample of predictors chosen. 

Boosting vs Bagging

Below is a video that I found helpful about understanding bagging and decision trees. The coursera course the video is a part of may be helpful as well.

[SAS]. Bagging [Video File]. Retrieved from [www.coursera.org/lecture/machine-learning-sas/bagging-Rj2Fq](https://www.coursera.org/lecture/machine-learning-sas/bagging-Rj2Fq) 

Coursera: Bagging using Decision Trees

Step 1: Here you replace the original data with new data. The new data usually have a fraction of the original data's columns and rows, which then can be used as hyper-parameters in the bagging model.

Step 2: You build classifiers on each dataset. Generally, you can use the same classifier for making models and predictions.

Step 3: Lastly, you use an average value to combine the predictions of all the classifiers, depending on the problem. Generally, these combined values are more robust than a single model.

Steps of Bagging

While bagging improves the prediction accuracy of decision trees, it reduces model interpretability because the ensemble can no longer be represented as a single tree. Variable importance measures address this by calculating the significance of predictors across all $$B$$ bagged trees. For regression trees, a variable's importance is determined by the total amount the Residual Sum of Squares (RSS) is decreased due to splits over that predictor, averaged over all $$B$$ trees. For classification trees, importance is measured by the total decrease in the Gini index caused by splits over the specific predictor, averaged across all $$B$$ trees. A larger average decrease indicates a more important predictor.

Learn Before

Related

Learn After