Starts with any simple model, then creates a new model based on the errors of the previous model, repeats till convergence. Iteratively makes a better model, by accounting for the errors of the predecessor model. At each stage you have created several different model. One thing to note here is that you simply dont take the model from the last stage but weighted sum of all models created before.

Boosting creates several trees (models) iteratively

D'Souza, Jocelyn. A Quick Guide to Boosting in ML. (2018, March). Retrieved from [medium.com/greyatom/a-quick-guide-to-boosting-in-ml-acf7c1585cb5](https://medium.com/greyatom/a-quick-guide-to-boosting-in-ml-acf7c1585cb5) 

A Quick Guide to Boosting in ML.

When nothing works, Boosting does. Nowadays many people use either XGBoost or LightGBM or CatBoost to win competitions at Kaggle or Hackathons. AdaBoost is the first stepping stone in the world of Boosting.
AdaBoost is one of the first boosting algorithms to be adapted in solving practices. Adaboost helps you combine multiple “weak classifiers” into a single “strong classifier”.

AdaBoosting 

Boosting algorithms are methods to help transform weak learners to stronger learners. There are 3 common boosting algorithms used in data science:

1. AdaBoosting (Adaptive Boosting)
2. GradientBoosting
3. XGBoost (Extreme Gradient Boosting)

University of Michigan - Ann Arbor

- GAM
- Boosting
- Support vector machines

Modern SL Methods (examples)

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Generalized additive models (GAMs) provide a general framework for
generalized additive model extending a standard linear model by allowing non-linear functions of each of the variables, while maintaining additivity. 

Generalized Additive Models

Common Boosting algorithm

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

There are 3 tuning parameters when using Boosting:
1) The first is the number of trees, denoted by $B$. Cross-validation is used in order to select $B$. If the value of $B$ is too big, it may overfit the data, albeit slowly. 
2) The second tuning parameter is the shrinkage parameter ($\lambda$), which controls the rate at which the boosting learns. This value is typically a small positive value; if it is very small, a large $B$ value may be required for it to work well.
3) The third tuning parameter is $d$, which is the number of splits in each tree. This value controls how complex the boosting will be. If $d = 1$, there is a single split, where each model has at most 1 variable (also known as a stump). It is also known as the ‘interaction depth’, which controls the interaction order. 


Tuning Parameters in Boosting

Boosting is heavily dependent on the trees formed before it, whereas in bagging, each tree is formed independently and varies instead based on the sample of predictors chosen. 

Boosting vs Bagging

Below is a video that I found helpful in understanding boosting and how its used with decision trees. The coursera course the video is a part of may be useful as well.

[SAS]. Boosting [Video File]. Retrieved from [www.coursera.org/lecture/machine-learning-sas/boosting-2J0Df](https://www.coursera.org/lecture/machine-learning-sas/boosting-2J0Df) 

Coursera: Boosting with Decision Trees

https://quantdare.com/what-is-the-difference-between-bagging-and-boosting/

what is boosting

Below is a representation of under and overfitting of the boundaries of U.S. sates. The data comes from the US Census Bureau. In its original format, the data is a single Keyhole Markup Language (KML) file which contains latitude and longitude coordinates of the borders of US states. The necessary latitude, longitude, and label (state) data were parsed from the KML files using a simple Python script. The main idea here is to understand the bias-variance trade-off and how that relates to under and overfitting. 

Additionally there are two examples of ways they avoided under and overfitting and created a much more accurate map using (gradient) boosting and random forest classifiers.

Learn Before

Related

Learn After