Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $B$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $b$th bootstrapped training data set to train the method to produce  $\hat{f}^{*b}(x)$. Afterwards, all the predictions are then averaged in order to get  $\hat{f}_{bag}(x) =  \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$. 

Bagging

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Weakness of Trees model is predictive accuracy compared to other models. We can tackle this weakness of Trees by:

 - Bagging
 - Random Forrests
 - Boosting

University of Michigan - Ann Arbor

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

!!For Regression Trees!!

For number of regions we decided to create for our predictor space
we find the splits that minimizes RSS for each region. 

This makes sense because we want to make each region as homogeneous as possible. 

However, considering all cases of splits for our predictor space is computationally infeaseable. There are simply too many cases to consider

As a solution to above issue we use 'recursive binary splitting methods' A.K.A Top down, greedy approach. 

TOP DOWN - Begin at the top of the tree and then successively split the predictor space

GREEDY - at each step of the tree-building process, the BEST split is made 
only considering at that specific step, rather than considering all future possible combinations of splits. 

!! For Classification Trees !!

Minimize Gini index to maximize node purity 

If there are K number of classes, Gini index is measure of total variance across the K classes. A small value indicates that a node contains predominantly observations from a single class

Additional Methods 
- Cross-enthropy : numerically very similar to Gini index.


How to segment predictor space Into Tree



Assume you have a decision boundary represented by two colors in below picture. If the true decision boundary is linear you can see that boxes created by trees are having hard time fitting the true decision boundary and vice versa for linear models when true decision boundary is non linear. 

Thus choosing between Tree and Linear models must be based on case by case analysis. 


Trees VS Linear Models 

How to improve the weakness of Trees model

The learning process of decision tree is essentially the process of selecting appropriate features, splitting and constructing tree nodes; The criterion of splitting node is determined by the information gain, information gain ratio and Gini coefficient before and after tree construction. These indicators are not related to the size of the current eigenvalue itself.

Learn Before

Related

Learn After