https://blogs.sas.com/content/iml/2018/12/12/essential-guide-bootstrapping-sas.html

University of Michigan - Ann Arbor

Bootstrap resampling is a way to create new samples of the original data. It obtains new datasets by sampling observations from the original set through replacement. These new datasets have the same sample size as the original and are useful in determining estimates for the standard error and confidence intervals of a statistic by comparing the accuracy of the model for each new data set.

Bootstrap method for resampling

Image Reference: Bootstrap

To create the bootstrap sample, select a random observation from your main data source, and then add the observation to your bootstrap sample. The observation then gets put back into your main data source (this is what allows observations to be selected more than once). This process gets repeated $n$ times until your bootstrap sample is complete.

Bootstrap resampling algorithm

Instead of repeatedly obtaining new data sets from a population, replacement provides an alternative to do just that from an already existing data set. Data sets made through replacement are limited to containing any points in the original data set. Some of the points from the original data set will show up multiple times, while others not at all.

Replacement in Bootstrap method for resampling

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $B$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $b$th bootstrapped training data set to train the method to produce  $\hat{f}^{*b}(x)$. Afterwards, all the predictions are then averaged in order to get  $\hat{f}_{bag}(x) =  \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$. 

Learn Before

Related