Graphical representation of using the decrease in Gini Index to determine the most important variables. 

University of Michigan - Ann Arbor

One downside of bagging is the tradeoff between prediction accuracy and model interpretability. Although bagging increases prediction accuracy, it does so at the expense of interpretability. We can no longer represent a bagged model as a single tree. It is also no longer clear which variables are most important to the model once bagging has been integrated. 

However, there are ways to determine the most important predictors. For regression trees, we can record the total amount the RSS is decreased as a result of the splits for a specific predictor. This value is then averaged over all $B$ trees and the predictor is considered important if that value is large. Likewise, with classification trees, we can record the total amount that the Gini index is decreased as a result of the splits for a specific predictor, and average it over $B$ trees.

Variable Importance Measures

Cite helpful visuals for helping explain data science concepts

Helpful data science images

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Gini index in information Theory

Image Reference: Variable Importance

https://www.textbook.ds100.org/ch/15/bias_cv.html



Image Reference: Train Test Error

Image Reference: Linear vs Logistic Regression

Link: [kaggle.com](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding) 

One hot encoding picture

Link: [runestone.academy](https://runestone.academy/runestone/books/published/httlads/_images/DSPipeline1.svg) 

Image Reference: Data Science Pipeline

https://manisha-sirsat.blogspot.com/2019/04/confusion-matrix.html

Image Reference: Confusion Matrix

https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5

ROC & AUC

https://www.intermediait.com/blog/2019/11/18/evaluating-model-performance-generalization-bias-variance-tradeoff-and-overfitting-vs-underfitting/

Workflow to reduce BIAS and VARIANCE

Recent Developments on Statistical and Neural Network Tools Focusing on Biodiesel Quality - Scientific Figure on ResearchGate. Available from: https://www.researchgate.net/figure/TOPOLOGY-OF-A-MULTILAYER-PERCEPTRON-NEURAL-NETWORK_fig1_274240858 [accessed 12 Feb, 2021]

Topology of a multilayer perceptron neural network

https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10

Learning Rate in Deep Learning

PART OF SPEECH TAGGER FOR ARABIC TEXT BASED SUPPORT VECTOR MACHINES: A REVIEW - Scientific Figure on ResearchGate. Available from: https://www.researchgate.net/figure/Methods-for-implementing-POS-tagger_fig1_331684946

Methods for implementing POS tagger


https://www.sentisum.com/library/nlp-and-text-mining

NLP and other relationships

https://medium.com/@tomyuz/a-sentiment-analysis-approach-to-predicting-stock-returns-d5ca8b75a42

Learn Before

Related