https://blogs.sas.com/content/iml/2018/12/12/essential-guide-bootstrapping-sas.html

Image Reference: Bootstrap

To create the bootstrap sample, select a random observation from your main data source, and then add the observation to your bootstrap sample. The observation then gets put back into your main data source (this is what allows observations to be selected more than once). This process gets repeated $n$ times until your bootstrap sample is complete.

Bootstrap resampling algorithm

Instead of repeatedly obtaining new data sets from a population, replacement provides an alternative to do just that from an already existing data set. Data sets made through replacement are limited to containing any points in the original data set. Some of the points from the original data set will show up multiple times, while others not at all.

Replacement in Bootstrap method for resampling

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $B$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $b$th bootstrapped training data set to train the method to produce  $\hat{f}^{*b}(x)$. Afterwards, all the predictions are then averaged in order to get  $\hat{f}_{bag}(x) =  \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$. 

Bagging

Bootstrap resampling is a way to create new samples of the original data. It obtains new datasets by sampling observations from the original set through replacement. These new datasets have the same sample size as the original and are useful in determining estimates for the standard error and confidence intervals of a statistic by comparing the accuracy of the model for each new data set.

University of Michigan - Ann Arbor

Resampling methods in supervised statistical learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Bootstrap method for resampling

Divides the training set into two parts - a training set and a validation set (also known as the “holdout set”). A model is made to fit the training set, and then the validation set is used to test the accuracy of the model because the model is blind to it. This gives an estimate of the test error of the model and can be useful in determining the best model from a series of models using different hyperparameters.

Learn Before

Related

Learn After