$$\delta(x)$$ is the difference of A and B on the initial test set. 

$$\delta(x^{(i)}$$ is the difference of A and B on the other test sets, which are created by resampling the initial test set. The total number of test sets is $$b$$.

$$H_{0} $$ is the null hypothesis, meaning that A is not better than B.


University of Colorado at Boulder

In NLP, it's common to use non-parametric tests, such as the Bootstrap test. Basically, the Bootstrap test means repeatedly drawing large numbers of smaller samples with replacement.

Let's say, we have a large number $$b$$ of virtual test sets $$x^{(i)}$$

$$
P = \sum^b_{i=1} $$𝟙$$ \left ( \delta(x^{(i)}) \geq 2\delta(x)  \right )
$$

if the p-value is smaller than 0.01, we can reject the null hypothesis and conclude A is better than B.

The Paired Bootstrap Test

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Learn Before

Related