$$\delta (x)$$ is the performance difference between A and B on a particular test set x. This performance can be accuracy or F-measure. A bigger δ means that A seems to be way better than B; a small δ means A seems to be only a little better.

$$\delta (X)$$ is the performance difference between A and B on X, which ranges over all test sets.

$$H_{0}$$ is the null hypothesis, meaning that A is not better than B.

p-value is the probability, assuming the null hypothesis $$H_{0}$$ is true, of seeing the δ (x) that we saw or one even greater.


Interpretation for Statistical Significance Test Formula

To compare the performance of classifiers of A and B:

$$P\left ( \delta  (X ) \geq  \delta  (x)|H_{0}\displaystyle \ is \ true \right ) $$

A very small p-value (smaller than 0.05 or 0.01) indicates that we can reject the null hypothesis. Thus, the result that A is better than B is statistically significant.



University of Colorado at Boulder

Statistical significance tests should be used to determine whether we can be confident that one version of a classifier is better than another.
It also can be used to determine which version of Naive Bayes classifier is better.

Statistical Significance Tests for Naive Bayes Classifier

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Formula of Statistical Significance Tests

In NLP, it's common to use non-parametric tests, such as the Bootstrap test. Basically, the Bootstrap test means repeatedly drawing large numbers of smaller samples with replacement.

Let's say, we have a large number $$b$$ of virtual test sets $$x^{(i)}$$

$$
P = \sum^b_{i=1} $$𝟙$$ \left ( \delta(x^{(i)}) \geq 2\delta(x)  \right )
$$

if the p-value is smaller than 0.01, we can reject the null hypothesis and conclude A is better than B.

Learn Before

Related

Learn After