$$\delta(x)$$ is the difference of A and B on the initial test set. 

$$\delta(x^{(i)}$$ is the difference of A and B on the other test sets, which are created by resampling the initial test set. The total number of test sets is $$b$$.

$$H_{0} $$ is the null hypothesis, meaning that A is not better than B.


Interpretation of Paired Bootstrap Test

In NLP, it's common to use non-parametric tests, such as the Bootstrap test. Basically, the Bootstrap test means repeatedly drawing large numbers of smaller samples with replacement.

Let's say, we have a large number $$b$$ of virtual test sets $$x^{(i)}$$

$$
P = \sum^b_{i=1} $$𝟙$$ \left ( \delta(x^{(i)}) \geq 2\delta(x)  \right )
$$

if the p-value is smaller than 0.01, we can reject the null hypothesis and conclude A is better than B.

University of Colorado at Boulder

Statistical significance tests should be used to determine whether we can be confident that one version of a classifier is better than another.
It also can be used to determine which version of Naive Bayes classifier is better.

Statistical Significance Tests for Naive Bayes Classifier

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

To compare the performance of classifiers of A and B:

$$P\left ( \delta  (X ) \geq  \delta  (x)|H_{0}\displaystyle \ is \ true \right ) $$

A very small p-value (smaller than 0.05 or 0.01) indicates that we can reject the null hypothesis. Thus, the result that A is better than B is statistically significant.



Learn Before

Related

Learn After