https://www.displayr.com/machine-learning-pruning-decision-trees/

Pruning Decision Trees Reference

Pruning decision trees is the process of removing certain branches which are not beneficial towards the model’s goal. Decision trees frequently over-fit, which leads to an excess of branches. As a result, a decision tree performs extremely well on the training set, but is over-fitting leading to poor test set performance. To prevent this, individuals can ‘prune’ the decision tree leading to fewer branches preventing over-fitting. Types of pruning include minimum error, smallest tree, and early stopping.

University of Michigan - Ann Arbor

Approaches for improving decision trees' predictions 

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $B$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $b$th bootstrapped training data set to train the method to produce  $\hat{f}^{*b}(x)$. Afterwards, all the predictions are then averaged in order to get  $\hat{f}_{bag}(x) =  \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$. 

Bagging

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Learn Before

Related

Learn After