It is useful and necessary to talk about conditional entropy, instead of leaving the subject where the textbook drops off. 
Let $$Y$$ be the set of labels, and let $$X$$ be the set of features. 

**Conditional Entropy** 
We now define conditional entropy as $$H(Y \mid X=x) = -\sum_{y\in Y} \Pr(Y=y)\log \Pr(Y=y \mid X=x)$$. 

We interpret the conditional entropy as the variability in the label, given the feature $$X=x$$. In context: how uncertain are we about a person’s health, given that we know how many times a year they go to the doctors office. 


Conditional Entropy

Similar to the Gini index numerically, entropy reflects node purity in a classification tree. A small value implies that one class dominates the region. Entropy is defined as: $$ D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k} $$ where $$\hat{p}_{m k}$$ denotes the proportion of training observations in the $$m$$th region that belong to the $$k$$th class.

University of Michigan - Ann Arbor

Google

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Gini index in information Theory

In the context of decision trees, the classification error rate is the fraction of training observations in a given region that do not belong to the most common class. It is mathematically defined as: $$ E = 1 - \max_{k} \left(\hat{p}_{m k}\right) $$ where $$\hat{p}_{m k}$$ represents the proportion of training observations in the $$m$$th region that belong to the $$k$$th class. While classification error rate is a natural metric, it is not sufficiently sensitive for tree-growing (splitting), making other measures like Gini index or entropy preferable in practice.

Learn Before

Related

Learn After