It is useful and necessary to talk about conditional entropy, instead of leaving the subject where the textbook drops off. 
Let $$Y$$ be the set of labels, and let $$X$$ be the set of features. 

**Conditional Entropy** 
We now define conditional entropy as $$H(Y \mid X=x) = -\sum_{y\in Y} \Pr(Y=y)\log \Pr(Y=y \mid X=x)$$. 

We interpret the conditional entropy as the variability in the label, given the feature $$X=x$$. In context: how uncertain are we about a person’s health, given that we know how many times a year they go to the doctors office. 


Conditional Entropy

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


University of Michigan - Ann Arbor

Two metrics in information Theory

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Gini index in information Theory

Entropy in information Theory

Classification error rate is the most natural alternative. It is defined as the fraction which represents the proportion of the training observations in the region that are not in the most common class:
$$
E=1-\max _{k}\left(\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.

*Classification error is not sufficiently sensitive for tree-growing, and in practice the two other measures are preferable.

Learn Before

Related

Learn After