Bag-of-Word Assumption is used to simplify the computation of Naive Bayes Classifier. It assumes that the positions of words in a document does not affect its classification. Based on this assumption, the features of a document can be simplified as the identical word encoding, discarding their relative ordering. Hence, the likelihood function P(d|c) can be written as $P(f_1,...,f_n|c)$

New York University

Naive Bayes Classifier is a probabilistic model. It implements the idea of Bayes Inference and works on two assumptions: Bag-of-Word Assumption and Naive Bayes Assumption.

Navie Bayes Classifier

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Bayes Inference derives the posterior probability based on a prior probability and a likelihood function according to Bayes' Theorem.
$P(C|D)=\frac{P(D|C)P(C)}{P(D)}$
Here, the probability $P(D|C)$ is the likelihood function, and $P(C)$ is the prior probability.

Bayesian Inference

The Naive Bayes classifier assigns the most probable class $$\hat{c}$$ by choosing the class which has the highest probability for a document $$d$$: $$\hat{c}=argmax_{c\in C}P(c|d)$$. Using Bayesian inference, this is equivalent to finding: $$\hat{c}=argmax_{c\in C}P(d|c)P(c)$$, where $$P(d|c)$$ is the likelihood function and $$P(c)$$ is the prior probability.

Bayesian Inference in Naive Bayes Classifier

Naive Bayes Classifier can be viewed as a generative model. Based on:
$\hat{c}=argmax_{c\in C}P(d|c)P(c)$, the generation of a document can be stated as: first, sample a class c from P(c); then, the words in the document can be generated by sampling from P(d|c).

Naive Bayes Classifier as a Generative Model

Bag of Word Assumption

Naive Bayes Assumption is the conditional independence assumption that is used to simplify the computation of the likelihood function $P(f_1,...,f_n|c)=P(f_1|c)...P(f_n|c)$.

Naive Bayes Assumption

The general model of Naive Bayes Classifier can be written as:
$\hat{c}=argmax_{c\in C}\log P(c)+\sum_i \log P(w_i|c)$,
where $C$ is the set of all possible classes, $i$ is the position of word $w_i$ in the text.
This is a linear model, since the prediction is computed as a linear combination of input features, ie., the words.

The General Expression of Naive Bayes Classifier for Text Classification

Sentiment Analysis can be viewed as a classification task with two classes, that is, positive and negative. Naive Bayes Classifier can be modified as Binary Multinomial Naive Bayes Classifier to improve its performance on sentiment analysis.

Application of Naive Bayes Classifier in Sentiment Analysis

We use a sentiment analysis domain with the two classes positive
(+) and negative (-), and take the following miniature training and test documents simplified from actual movie reviews.

Learn Before

Related