The accuracy of object recognition has dramatically increased due to deep learning architectures. For example, on the ImageNet benchmark, the top-five classification error rate plummeted from $$28\%$$ in 2010 to $$2.25\%$$ by 2017. Beyond benchmark datasets, this improved visual perception enables complex real-world applications such as diagnosing skin cancer from medical images and identifying birdsong.

Object Recognition Benchmark Improvements

Classification error rate is the most natural alternative. It is defined as the fraction which represents the proportion of the training observations in the region that are not in the most common class:
$$
E=1-\max _{k}\left(\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.

*Classification error is not sufficiently sensitive for tree-growing, and in practice the two other measures are preferable.

University of Michigan - Ann Arbor

Claude

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Dive into Deep Learning

Classification error rate

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


Entropy in information Theory

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Learn Before

Related

Learn After