To compare the performance of classifiers of A and B:

$$P\left ( \delta  (X ) \geq  \delta  (x)|H_{0}\displaystyle \ is \ true \right ) $$

A very small p-value (smaller than 0.05 or 0.01) indicates that we can reject the null hypothesis. Thus, the result that A is better than B is statistically significant.



Formula of Statistical Significance Tests

In NLP, it's common to use non-parametric tests, such as the Bootstrap test. Basically, the Bootstrap test means repeatedly drawing large numbers of smaller samples with replacement.

Let's say, we have a large number $$b$$ of virtual test sets $$x^{(i)}$$

$$
P = \sum^b_{i=1} $$𝟙$$ \left ( \delta(x^{(i)}) \geq 2\delta(x)  \right )
$$

if the p-value is smaller than 0.01, we can reject the null hypothesis and conclude A is better than B.

The Paired Bootstrap Test

Statistical significance tests should be used to determine whether we can be confident that one version of a classifier is better than another.
It also can be used to determine which version of Naive Bayes classifier is better.

University of Colorado at Boulder

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

(Naive) Bayes Classifier

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

https://medium.com/@seevee/machine-learning-bayes-theorem-2f48c33d51e5

Bayes Theorem in Machine Learning

 $$P(Y=j|X=x_{0}) = (P(X=x_{0}|Y=j)\times P(Y=j))/P(X=x_{0}) $$
This is an expanded equation.

$x_{0}$: predictor vector
j: class
Probability of the response (class j) given the input (predictor vector) for each observation.

This equation can be used with relational comparisons with fixed probability values to place observations into classes.


https://machinelearningmastery.com/bayes-theorem-for-machine-learning/

Bayes Formula

"This study presents the applicability of the Naïve Bayes classifier ensemble for traffic incident detection. The standard Naive Bayes (NB) has been applied to traffic incident detection and has achieved good results. However, the detection result of the practically implemented NB depends on the choice of the optimal threshold, which is determined mathematically by using Bayesian concepts in the incident-detection process."

Qingchao Liu, Jian Lu, Shuyan Chen, Kangjia Zhao, "Multiple Naïve Bayes Classifiers Ensemble for Traffic Incident Detection", Mathematical Problems in Engineering, vol. 2014, Article ID 383671, 16 pages, 2014. https://doi.org/10.1155/2014/383671

Article: Multiple Naïve Bayes Classifiers Ensemble for Traffic Incident Detection

The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

Pros:
- Easy to understand
- Highly efficient learning and prediction
- Works well with high-dimensional data
- Used as a baseline comparison

Cons:
- Generalization is worse compared to more sophisticated models
- The conditional independence assumption doesn't always hold
- Low accuracy of the confidence estimates for predictions.

Pros and Cons of Naive Bayes Classifier

While training, we first consider maximum likelihood estimate.
We first concatenate all documents with category c into one big “category c” text. Then we use the frequency of wi in this concatenated document to give a maximum likelihood estimate of the probability:
$P(w_i|c) = \frac{count(w_i,c)}{\sum_{w\in V}count(w,c)}$

Training the naive Bayes Classifier

A naïve Bayes classifier using only individual word features of all of the words in a given text is similar to language modeling. Such a naïve Bayes model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model.

The model also assigns a probability to each sentence:

$ P(s|c) = \sum_{i\in positions}P(w_i|c) $

Naïve Bayes as a Language Model

Statistical Significance Tests for Naive Bayes Classifier

It is important to avoid harms that may result from classifiers.
$$ \\ $$
Representational harms: harms caused by a system that demeans a social group. For example, African American names are more likely to be assigned negative emotion in the sentiment analysis.
$$ \\ $$
Censorship harms: like in toxicity detection, false-positive errors could lead to the censoring of discourse about certain groups, like gay people and blind people.
$$ \\ $$
A model card for NLP can help to clear and find the harms:
A model card includes the following information:
• training algorithms and parameters
• training data sources, motivation, and preprocessing
• evaluation data sources, motivation, and preprocessing
• intended use and users
• model performance across different demographic or other groups and environmental situations



Avoiding Harms

- Naive Bayes can be used to represent any property in the input text.
- Combining linguistic and non linguistic features in the case of spam email detection.
- Language Id is considered as an input feature.

 Naive Bayes for other text classification tasks

The primary types of Naive Bayes classifiers are Bernoulli Naive Bayes, Multinomial Naive Bayes, and Gaussian Naive Bayes.

Learn Before

Related

Learn After