An improvement proposed by Ross Quinlan that corrects for the disadvantage of Information Gain by taking into account the number of values a feature can take. 
Information Gain Ratio (IGR) is just the ratio of IG and Intrinsic Value (IV) which is defined as :

$$IV(X, Y) = - \Sigma_{v \in values(Y)} \frac{|\{X_Y == v\}|}{|X|} \log_2 \frac{|\{X_Y == v\}|}{|X|}$$
Thus, Information Gain Ratio is : 

$$IGR(X,Y) = \frac{IG(X,Y)}{IV(X,Y)}$$


Information Gain Ratio

Information Gain tries to split trees on the basis of the most relevant feature X, however it does NOT take into account the number of values that X can take. If a feature A has a higher information gain and can take multiple distinct values that can exactly span the testing data, it can lead to overfitting. 

University of Michigan - Ann Arbor


We define information gain as $$IG(X,Y) = H(Y) - H(Y\mid X)$$. 

We interpret information gain as the amount of information we learn about the label $$Y$$, given a specific feature $$X$$. 

Note that if the feature $$X$$ is completely uncorrelated with the label $$Y$$, then $$H(Y\mid X) = H(Y)$$. Therefore $$IG(X,Y) = 0$$. 

As such when we choose the next feature to split on we should choose the one that maximizes information gained. 


Information Gain

Consider a binary classification problem with 29 positve labels and 35 negative. You are building a decision tree and are trying to figure out whether to split on feature A1 or A2. Given the following information, which feature is the better choice?  You should calculate IG as an exercise.


Learn Before

Related

Learn After