The learning process of decision tree is essentially the process of selecting appropriate features, splitting and constructing tree nodes; The criterion of splitting node is determined by the information gain, information gain ratio and Gini coefficient before and after tree construction. These indicators are not related to the size of the current eigenvalue itself.

University of Michigan - Ann Arbor

Normalization is one of the techniques used to reduce the bias and variance in statistical learning. It helps to have all the variables centered around 0 and have the same variance.
We subtract all the values of each variable from their mean and divide them by their variance.
$$\mu = \frac{1}{m} \sum_{i = 1}^{m} X^{(i)}$$
$$\sigma^{2} = \frac{1}{m} \sum_{i = 1}^{m} (X^{(i)} - \mu)^2$$
$$X_{New} = \frac{X - \mu}{\sigma}$$

Normalization of Data

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Normalization of training data may help reducing the bias and variance, and speeds up Gradient Descent, and consequently, training of the deep learning model, as shown in the figure.

Normalization Helps Deep Learning

Why do we normalize the inputs x?

Why not normalize the tree structure?

!!For Regression Trees!!

For number of regions we decided to create for our predictor space
we find the splits that minimizes RSS for each region. 

This makes sense because we want to make each region as homogeneous as possible. 

However, considering all cases of splits for our predictor space is computationally infeaseable. There are simply too many cases to consider

As a solution to above issue we use 'recursive binary splitting methods' A.K.A Top down, greedy approach. 

TOP DOWN - Begin at the top of the tree and then successively split the predictor space

GREEDY - at each step of the tree-building process, the BEST split is made 
only considering at that specific step, rather than considering all future possible combinations of splits. 

!! For Classification Trees !!

Minimize Gini index to maximize node purity 

If there are K number of classes, Gini index is measure of total variance across the K classes. A small value indicates that a node contains predominantly observations from a single class

Additional Methods 
- Cross-enthropy : numerically very similar to Gini index.


How to segment predictor space Into Tree



Assume you have a decision boundary represented by two colors in below picture. If the true decision boundary is linear you can see that boxes created by trees are having hard time fitting the true decision boundary and vice versa for linear models when true decision boundary is non linear. 

Thus choosing between Tree and Linear models must be based on case by case analysis. 


Trees VS Linear Models 

Weakness of Trees model is predictive accuracy compared to other models. We can tackle this weakness of Trees by:

 - Bagging
 - Random Forrests
 - Boosting

Learn Before

Related