**Paired bootstrap resampling** adapts general bootstrap resampling to compare two systems evaluated on the *same* set of items (e.g., the same test questions). Given $$n$$ paired per-item scores $$(a_i, b_i)$$ for systems $$A$$ and $$B$$ under the same metric, one draws $$B$$ resamples by sampling item indices with replacement; on each resample $$b$$, both systems' metrics are recomputed on exactly the same resampled indices and a paired delta $$\hat{\Delta}_b = M_A^{(b)} - M_B^{(b)}$$ is recorded. The empirical distribution of $$\{\hat{\Delta}_b\}_{b=1}^{B}$$ is used to form a **percentile confidence interval** for the true paired difference (e.g., the $$2.5$$th and $$97.5$$th percentiles for a 95% CI) and to derive a one-sided $$p$$-value as the fraction of resamples in which the delta has the opposite sign. A delta is treated as **significant** when its percentile CI excludes $$0$$. Pairing the resampled index set for both systems removes between-item variance and is the standard significance protocol for comparing two systems on a shared evaluation set in NLP.

Paired Bootstrap Resampling for Significance Testing

To create the bootstrap sample, select a random observation from your main data source, and then add the observation to your bootstrap sample. The observation then gets put back into your main data source (this is what allows observations to be selected more than once). This process gets repeated $$n$$ times until your bootstrap sample is complete.

University of Michigan - Ann Arbor

Google

Bootstrap resampling is a way to create new samples of the original data. It obtains new datasets by sampling observations from the original set through replacement. These new datasets have the same sample size as the original and are useful in determining estimates for the standard error and confidence intervals of a statistic by comparing the accuracy of the model for each new data set.

Bootstrap method for resampling

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

https://blogs.sas.com/content/iml/2018/12/12/essential-guide-bootstrapping-sas.html

Image Reference: Bootstrap

Instead of repeatedly obtaining new data sets from a population, replacement provides an alternative to do just that from an already existing data set. Data sets made through replacement are limited to containing any points in the original data set. Some of the points from the original data set will show up multiple times, while others not at all.

Replacement in Bootstrap method for resampling

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $B$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $b$th bootstrapped training data set to train the method to produce  $\hat{f}^{*b}(x)$. Afterwards, all the predictions are then averaged in order to get  $\hat{f}_{bag}(x) =  \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$. 

Learn Before

Related

Learn After