Two metrics in information Theory

In order to determine the best feature to split on next when building a decision tree, we need to have some measure of “information gain” for each feature. To do this, we consider two metrics.


University of Michigan - Ann Arbor

The algorithm required to build a decision tree involves 3 steps:

1. Start with an empty tree

2. Split on the feature with the highest information gain

3. Recurse with step 2

Creating a minimally sized tree which minimizes training error is actually an extremely hard, (NP-Hard) problem). To approximate this optimal solution, we build a tree by greedily splitting on the feature which results in the most information gain, i.e., pick the feature to split on that more certainly leads to a classification. 


Learning Decision Trees

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

Learn Before

Related

Learn After