```python
from sklearn.externals.six import StringIO  
from IPython.display import Image  
from sklearn.tree import export_graphviz
import pydotplus

#split into train and test 
df_iris_train,df_iris_test = train_test_split(df_iris, petal_width > 5.5)

#build and train the classifier
dt_model = sktree.DecisionTreeClassifier(max_depth=1,
                                         criterion='entropy')
#given first 4 columns, learn the species
dt_model.fit(df_iris_train.ix[:,:4],df_iris_train.species)

dot_data = StringIO()
export_graphviz(dt_model, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
Image(graph.create_png())

#Deploying a decision tree
#this is testing the model 
predicted_labels = dt_model.predict(df_iris_test.ix[:,:4])
df_iris_test['predicted_label_tree'] = predicted_labels
df_iris_test.sample(10)

#Determine the overall accuracy
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(df_iris_test.species, predicted_labels)
print("Accuracy: {0:.2%}".format(accuracy))

#utility class to compare the predictions versus ground truth
def comparePlot(input_frame,real_column,predicted_column):
    df_a = input_frame.copy()
    df_b = input_frame.copy()
    df_a['label_source'] = 'Species'
    df_b['label_source'] = 'Classifier'
    df_a['label'] = df_a[real_column]
    df_b['label'] = df_b[predicted_column].apply(lambda x: 'Predict %s'%x)
    df_c = pd.concat((df_a, df_b), axis=0, ignore_index=True)
    sns.lmplot(x='sepal_length', y='sepal_width', col='label_source',
               hue='label', data=df_c, fit_reg=False, size=4);

#see the difference
comparePlot(df_iris_test,"species","predicted_label_tree")
```


Python Code for Classification and Decision Tree

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

Classification Tree is used to predict qualitative variables instead of quantitative variables. It is very similar to regression tree and use recursive binary splitting to generate the classification tree. But use *Gini index* 
$$ G = \sum_{k=1}^{K} \hat{p}_{mk} (1-\hat{p}_{mk}) $$
and *entropy* as the indicator. 
$$ D = -\sum_{k=1}^{K} \hat{p}_{mk} log \hat{p}_{mk} $$

The small value of purity of node indicated that a node contains observations from almost the same category. *Entropy* is the same as Gini Index, if the $m$th node is pure, the *entropy* is very small. 

University of Michigan - Ann Arbor

Decision trees can be applied to both regression and classificatioin problems, and can thus be divided into

- Regression trees
- Classification trees

Decision trees applied to regression and classificatioin problems

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Roughly speaking, there are two steps.
1. We divide the predictor space—that is, the set of possible values for X1, X2, . . . , Xp—into J distinct and non-overlapping regions, R1,R2,...,RJ.
2. For every observation that falls into the region Rj , we make the same prediction, which is simply the mean of the response values for the training observations in Rj.

Steps of creating a decision tree

1. Over fitting -  Can create over-complex trees that do not generalize the data well. This problem gets solved by setting constraints on model parameters and pruning.
2. Not fit for continuous variables - While working with continuous numerical variables, decision tree loses information, when it categorizes variables in different categories.
3. Generally low accuracy compared to other ML algorithms
4.  Potentially unstable - because small variations in the data might result in a completely different tree being generated. This is called variance, which needs to be lowered by methods like bagging and boosting.

Source: [medium.com/greyatom/decision-trees-a-simple-way-to-visualize-a-decision-dc506a403aeb ](https://medium.com/greyatom/decision-trees-a-simple-way-to-visualize-a-decision-dc506a403aeb )

Decision Tree Disadvantages

The predictor space ($X_1, X_2, …, X_p$) is divided into J non-overlapping regions. Then make the same prediction for each observation that falls into the region $R_j$, and the predicted value is the mean of the response values for the training observations in $R_j$. The goal is to find the matrix region that minimizes the RSS of the model $\sum_{j=1}^{J} \sum_{i \in R_j} (y_i - \hat{y}_{R_j})^2$. 

Recursion Binary Splitting: from the top of the tree (*top-down*), the prediction space is split to two new branches, and the determination of the optimal (*best*) split is limited to a particular step instead of some future step. 

Cost Complexity pruning, as known as Weakest Link Pruning, is able to avoid overfitting. 
$$ \sum_{m=1}^{|T|} \sum_{i:x_i \in {R_m}} (y_i - \hat{y}_{R_m})^2 + \alpha |T| $$


Learn Before

Related

Learn After