Classification error rate is the most natural alternative. It is defined as the fraction which represents the proportion of the training observations in the region that are not in the most common class:
$$
E=1-\max _{k}\left(\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.

*Classification error is not sufficiently sensitive for tree-growing, and in practice the two other measures are preferable.

Classification error rate

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


Entropy in information Theory

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Gini index in information Theory

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

University of Michigan - Ann Arbor

Apply the decision tree method to regression problems. It is quite similar to regression trees except that a qualitative, not quantitive, response is needed. This leads to following differences:
* The criterion for making the binary splits is no longer RSS.
* The most commonly class in the region, rather than the mean value, is used as the prediction result. 
Classification Tree is used to predict qualitative variables instead of quantitative variables. It is very similar to regression tree and use recursive binary splitting to generate the classification tree. But use *Gini index* 
$$ G = \sum_{k=1}^{K} \hat{p}_{mk} (1-\hat{p}_{mk}) $$
and *entropy* as the indicator. 
$$ D = -\sum_{k=1}^{K} \hat{p}_{mk} log \hat{p}_{mk} $$

The small value of purity of node indicated that a node contains observations from almost the same category. *Entropy* is the same as Gini Index, if the $m$th node is pure, the *entropy* is very small. 

Classification trees

Classification Tree is used to predict qualitative variables instead of quantitative variables. It is very similar to regression tree and use recursive binary splitting to generate the classification tree. But use *Gini index* 
$$ G = \sum_{k=1}^{K} \hat{p}_{mk} (1-\hat{p}_{mk}) $$
and *entropy* as the indicator. 
$$ D = -\sum_{k=1}^{K} \hat{p}_{mk} log \hat{p}_{mk} $$

The small value of purity of node indicated that a node contains observations from almost the same category. *Entropy* is the same as Gini Index, if the $m$th node is pure, the *entropy* is very small. 

Classification Tree

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Criteria for making binary splits in a classification tree

Step 1: Change numerical variable into categorical variable
You have two options, 
if it's binary(this will add a new column):
newcolumnname=ifelse(columnname <=threshold,"No","Yes")
newdataset =data.frame(dataset ,newcolumnname) used to merge the column with original dataset

if it's multi-category(this will change original column):
data$column[criterion] <- 'category name1'
data$column[criterion] <- 'category name2'

Step 2: Build a model 

To build a decision tree,we use function tree( )
model<-tree(formula=outputcolumn~., data=dataframe)


Step 3: Predict on test data
tree.pred=predict(model ,testdata,type="class")
table(tree.pred,testlabel)

Step 4:Consider whether prune can lead to improved result
The function cv.tree() performs cross-validation in order to
determine the optimal level of tree complexity; We use the argument FUN=prune.misclass in order to indicate that we want the classiﬁcation error rate to guide the cross-validation and pruning process, rather than the default for the cv.tree() function, which is deviance.

cv.carseats =cv.tree(tree.carseats ,FUN=prune.misclass )





Implemention of Classification Tree in R

https://towardsdatascience.com/decision-tree-classification-de64fc4d5aac

More on classification and decision trees

```python
from sklearn.externals.six import StringIO  
from IPython.display import Image  
from sklearn.tree import export_graphviz
import pydotplus

#split into train and test 
df_iris_train,df_iris_test = train_test_split(df_iris, petal_width > 5.5)

#build and train the classifier
dt_model = sktree.DecisionTreeClassifier(max_depth=1,
                                         criterion='entropy')
#given first 4 columns, learn the species
dt_model.fit(df_iris_train.ix[:,:4],df_iris_train.species)

dot_data = StringIO()
export_graphviz(dt_model, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
Image(graph.create_png())

#Deploying a decision tree
#this is testing the model 
predicted_labels = dt_model.predict(df_iris_test.ix[:,:4])
df_iris_test['predicted_label_tree'] = predicted_labels
df_iris_test.sample(10)

#Determine the overall accuracy
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(df_iris_test.species, predicted_labels)
print("Accuracy: {0:.2%}".format(accuracy))

#utility class to compare the predictions versus ground truth
def comparePlot(input_frame,real_column,predicted_column):
    df_a = input_frame.copy()
    df_b = input_frame.copy()
    df_a['label_source'] = 'Species'
    df_b['label_source'] = 'Classifier'
    df_a['label'] = df_a[real_column]
    df_b['label'] = df_b[predicted_column].apply(lambda x: 'Predict %s'%x)
    df_c = pd.concat((df_a, df_b), axis=0, ignore_index=True)
    sns.lmplot(x='sepal_length', y='sepal_width', col='label_source',
               hue='label', data=df_c, fit_reg=False, size=4);

#see the difference
comparePlot(df_iris_test,"species","predicted_label_tree")
```


Learn Before

Related

Learn After