!!For Regression Trees!!

For number of regions we decided to create for our predictor space
we find the splits that minimizes RSS for each region. 

This makes sense because we want to make each region as homogeneous as possible. 

However, considering all cases of splits for our predictor space is computationally infeaseable. There are simply too many cases to consider

As a solution to above issue we use 'recursive binary splitting methods' A.K.A Top down, greedy approach. 

TOP DOWN - Begin at the top of the tree and then successively split the predictor space

GREEDY - at each step of the tree-building process, the BEST split is made 
only considering at that specific step, rather than considering all future possible combinations of splits. 

!! For Classification Trees !!

Minimize Gini index to maximize node purity 

If there are K number of classes, Gini index is measure of total variance across the K classes. A small value indicates that a node contains predominantly observations from a single class

Additional Methods 
- Cross-enthropy : numerically very similar to Gini index.


How to segment predictor space Into Tree



Assume you have a decision boundary represented by two colors in below picture. If the true decision boundary is linear you can see that boxes created by trees are having hard time fitting the true decision boundary and vice versa for linear models when true decision boundary is non linear. 

Thus choosing between Tree and Linear models must be based on case by case analysis. 


Trees VS Linear Models 

Weakness of Trees model is predictive accuracy compared to other models. We can tackle this weakness of Trees by:

 - Bagging
 - Random Forrests
 - Boosting

How to improve the weakness of Trees model

The learning process of decision tree is essentially the process of selecting appropriate features, splitting and constructing tree nodes; The criterion of splitting node is determined by the information gain, information gain ratio and Gini coefficient before and after tree construction. These indicators are not related to the size of the current eigenvalue itself.

Why not normalize the tree structure?

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


University of Michigan - Ann Arbor

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree-based Methods 

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Artificial Neural Networks are built to resemble neurons in the human brain. The idea behind neural networks is to understand the pattern behind terms through examples rather than through task-specific rules.

Artificial Neural Networks

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

Learn Before

Related

Learn After