https://medium.com/@seevee/machine-learning-bayes-theorem-2f48c33d51e5

Bayes Theorem in Machine Learning

 $$P(Y=j|X=x_{0}) = (P(X=x_{0}|Y=j)\times P(Y=j))/P(X=x_{0}) $$
This is an expanded equation.

$x_{0}$: predictor vector
j: class
Probability of the response (class j) given the input (predictor vector) for each observation.

This equation can be used with relational comparisons with fixed probability values to place observations into classes.


https://machinelearningmastery.com/bayes-theorem-for-machine-learning/

Bayes Formula

"This study presents the applicability of the Naïve Bayes classifier ensemble for traffic incident detection. The standard Naive Bayes (NB) has been applied to traffic incident detection and has achieved good results. However, the detection result of the practically implemented NB depends on the choice of the optimal threshold, which is determined mathematically by using Bayesian concepts in the incident-detection process."

Qingchao Liu, Jian Lu, Shuyan Chen, Kangjia Zhao, "Multiple Naïve Bayes Classifiers Ensemble for Traffic Incident Detection", Mathematical Problems in Engineering, vol. 2014, Article ID 383671, 16 pages, 2014. https://doi.org/10.1155/2014/383671

Article: Multiple Naïve Bayes Classifiers Ensemble for Traffic Incident Detection

The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

Pros:
- Easy to understand
- Highly efficient learning and prediction
- Works well with high-dimensional data
- Used as a baseline comparison

Cons:
- Generalization is worse compared to more sophisticated models
- The conditional independence assumption doesn't always hold
- Low accuracy of the confidence estimates for predictions.

Pros and Cons of Naive Bayes Classifier

While training, we first consider maximum likelihood estimate.
We first concatenate all documents with category c into one big “category c” text. Then we use the frequency of wi in this concatenated document to give a maximum likelihood estimate of the probability:
$P(w_i|c) = \frac{count(w_i,c)}{\sum_{w\in V}count(w,c)}$

Training the naive Bayes Classifier

A naïve Bayes classifier using only individual word features of all of the words in a given text is similar to language modeling. Such a naïve Bayes model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model.

The model also assigns a probability to each sentence:

$ P(s|c) = \sum_{i\in positions}P(w_i|c) $

Naïve Bayes as a Language Model

Statistical significance tests should be used to determine whether we can be confident that one version of a classifier is better than another.
It also can be used to determine which version of Naive Bayes classifier is better.

Statistical Significance Tests for Naive Bayes Classifier

It is important to avoid harms that may result from classifiers.
$$ \\ $$
Representational harms: harms caused by a system that demeans a social group. For example, African American names are more likely to be assigned negative emotion in the sentiment analysis.
$$ \\ $$
Censorship harms: like in toxicity detection, false-positive errors could lead to the censoring of discourse about certain groups, like gay people and blind people.
$$ \\ $$
A model card for NLP can help to clear and find the harms:
A model card includes the following information:
• training algorithms and parameters
• training data sources, motivation, and preprocessing
• evaluation data sources, motivation, and preprocessing
• intended use and users
• model performance across different demographic or other groups and environmental situations



Avoiding Harms

- Naive Bayes can be used to represent any property in the input text.
- Combining linguistic and non linguistic features in the case of spam email detection.
- Language Id is considered as an input feature.

 Naive Bayes for other text classification tasks

The primary types of Naive Bayes classifiers are Bernoulli Naive Bayes, Multinomial Naive Bayes, and Gaussian Naive Bayes.

Types of Naive Bayes Classifiers

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

University of Michigan - Ann Arbor

San Jose State University

Bayes Theorem is about predicting the probability of an event A occurring given an event B has already occurred. We can use the inverse probability of this statment and the probability of each event to find this value. 
P(A) is called prior probability, P(B|A) is Likelihood, P(B) is evidence, and P(A|B) is posterior probability.


Bayes Theorem Overview

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Augmented Startups. (2017, 26 August). Naive Bayes Classifier [Video]. Youtube. [www.youtube.com/watch?v=CPqOCI0ahss](https://www.youtube.com/watch?v=CPqOCI0ahss) 

YouTube: Augmented Startups - Naïve Bayes Classifier

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

(Naive) Bayes Classifier

According to Bayes' theorem, we have
$$
P\left(Y=k | X=x\right)=\frac{P\left(Y=k\right) P\left(X=x | Y=k\right)}{\sum_{l=1}^{K} P\left(Y=l\right) P\left(X=x | Y=l\right)}\\
=\frac{\pi_{k} f_{k}(x)}{\sum_{l=1}^{K} \pi_{l} f_{l}(x)}.
$$
In LDA with the assumption of Gaussian distribution in every class, we have the Gaussian density 
$$
f_k(x)=\frac{1}{(2 \pi_k)^{p / 2}|\boldsymbol{\Sigma}|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu_k)^{T} \boldsymbol{\Sigma}^{-1}(x-\mu_k)\right)
$$
for a $p$-dimensional random vector $X$ with the distribution $N(\mu_k, \mathbf{\Sigma})$. All we need to do is to plug $f_k(x)$ into Bayes' theorem above and find out the $k$ which maximize $P\left(Y=k | X=x\right)$. LDA is a Bayes classifier with the assumption of Gaussian distribution in every class.

Using Bayes' theorem in LDA

Bayesian networks were developed to understand reasoning under conditions of uncertainty. It is based on Bayes Theorem and is a probabilistic model that represents a set of variables and their conditional dependencies by being able to find the probability of facts being true/false given already defined facts. A Bayesian network is represented by a directed acyclic graph which helps to visualize the probabilistic model for reviewing relationships between random variables and reasonings for causal probabilities given certain facts



Bayesian Networks

Example 1: Imagine a billiard table, and suppose that the billiard track bounces off the table many times, so we don't know where the ball will end up. Let's say the billiard table is L, and the probability that the ball ends up x feet from the edge is x/L.

The inverse probability problem: Suppose we observe that the ball's final stop is a foot away from the table, but we don't know the length of the table, L.

Bayes' main work is to break this cognitive asymmetry and propose a method to estimate the inverse probability.


Inverse-probability: break cognitive asymmetry 

This example in the book is a discussion on how we can update our beliefs using Bayes's rule.  The question the example tries to address is, "What is the probability that she has breast cancer, given that the test came out positive?", which can be expressed as P(D | T), where D is the hypothesis, disease, and T is the evidence, test.

$$
        $$(Updated probability of D) = P(D | T) = (likelihood ratio)) × (prior probability of D)$$
      $$

(Note) The "likelihood ratio" is given by P(T | D)/P(T). 



Mammogram (Breast Cancer Screening Example)

There are three out of four binary classification metrics mentioned in the Book of Why; true positive, false positive, and false negative.  The fourth one that is not mentioned in the book is true negative.  The following are the two primary building blocks of such four metrics: 

1. True vs. False: These are about prediction's correctness whether a prediction outcome was made accurately/ correctly.  The correct prediction is True, and the incorrect prediction is False.
2. Positive vs. Negative: These are about a binary classification -(e.g.) In the mammogram example in the book, "positive" is positive test results (breast cancer), whereas "negative" is negative test results (not breast cancer).  

The combinations of the above concepts result in True positive, True negative, false positive, and false negative, each of which is explained in the subsequent nodes to follow.  


Binary Classification Metrics

Inverse probability, as supposed to forward probability, is a conditional probability that a hypothesis (cause), H, is true given that the evidence (effect), E is true.  Inverse probabiity is harder to estimate than estimating a forward probability.  

It can be expressed as:

	 $$ P(H | E) $$


Inverse Probability

Forward probability, as supposed to inverse probability, is the conditional probability that an evidence (effect), E, is true given that the hypothesis (cause), H is true.  For the definition of inverse probability, see the node, "Inverse Probability Defined." 

It can be expressed as:

  $$P(E | H)$$

Estimating the forward probability is easier than estimating the inverse probability because, as long as we know the cause, it is easy to estimate the probability of the effect.  

Pearl calls forward probability, calculating conditional probability in the direction, in which "our judgment is more reliable."


Forward Probability

Bayes' Theorem can also be used as the following :

$$ P(A | B) = \frac{P(A \cap B)}{P(B)}$$
The way to think about this is that given two events A and B, the probability that A happens, given B occurs, is the probability of their intersection occuring i.e. both events occuring, over the probability that B occurs.
The easiest way to see why this works is for Independent events. If A and B are independent, the probability that they both occur is the product of their individual probabilities. Which then simplifies the above equation to :
$$ P(A|B)    =    \frac{P(A \cap B)}{P(B)}$$
$$ \implies P(A|B) = \frac{P(A).P(B)}{P(B)}$$
$$ \implies P(A|B) = P(A)$$
Which makes sense since the events are independent.

Alternate Bayes' Theorem Equation

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Artificial Neural Networks are built to resemble neurons in the human brain. The idea behind neural networks is to understand the pattern behind terms through examples rather than through task-specific rules.

Learn Before

Related

Learn After