Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


University of Michigan - Ann Arbor

Two metrics in information Theory

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

One downside of bagging is the tradeoff between prediction accuracy and model interpretability. Although bagging increases prediction accuracy, it does so at the expense of interpretability. We can no longer represent a bagged model as a single tree. It is also no longer clear which variables are most important to the model once bagging has been integrated. 

However, there are ways to determine the most important predictors. For regression trees, we can record the total amount the RSS is decreased as a result of the splits for a specific predictor. This value is then averaged over all $B$ trees and the predictor is considered important if that value is large. Likewise, with classification trees, we can record the total amount that the Gini index is decreased as a result of the splits for a specific predictor, and average it over $B$ trees.

Variable Importance Measures

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Gini index in information Theory

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


Entropy in information Theory

Classification error rate is the most natural alternative. It is defined as the fraction which represents the proportion of the training observations in the region that are not in the most common class:
$$
E=1-\max _{k}\left(\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.

*Classification error is not sufficiently sensitive for tree-growing, and in practice the two other measures are preferable.

Classification error rate

Graphical representation of using the decrease in Gini Index to determine the most important variables. 

Learn Before

Related