Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Gini index in information Theory

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


Entropy in information Theory

University of Michigan - Ann Arbor

In order to determine the best feature to split on next when building a decision tree, we need to have some measure of “information gain” for each feature. To do this, we consider two metrics.


Intro to Information Theory

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

Learn Before

Related

Learn After