Pros:
- With GAMS, one can fit non-linear functions to each predictor to automatically model non-linear relationships that standard linear regression will miss
- The non-linear fits *can* make more accurate predictions for the response Y
- The additive nature of GAMs means we can still examine the effect of each predictor variable on Y while holding other variables fixed.
- Degrees of freedom can summarize the smoothness of the GAM

Cons:
- The main limitation is that GAMs are restricted to be additive, meaning important interactions *can be missed*. However, as with regular linear regression, we can manually add interactions by including additional predictors.













Pros and Cons of GAMS

Example with wage data: 

Since this is simply a linear regression model with several basis functions, we simply use the lm() function.

```R 
gam1 <- lm(wage ~ ns(age, 5) + education, 
           data=Wage)
```

To fit more complex splines, or other components that cannot be specified as basis functions, we need to use the gam library in R. the s() function is used to indicate a smoothing spline.

```R
library(gam)

## Loaded gam 1.09.1
gam2 <- gam(wage~s(year, 4) + s(age, 5) + education, 
            data=Wage)
```

R implementation of GAMs

For Generalized Additive Models (GAMs) in regression problems, to depict a non-linear relationship between predictors and their response, we start with the multiple linear regression model $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \dots + \beta_px_{ip} + \epsilon_i$$. We then replace each linear component ($$\beta_jx_{ij}$$) with a non-linear function $$f_j(x_{ij})$$ so that the model becomes $$y_i = \beta_0 + f_1(x_{i1}) + f_2(x_{i2}) + \dots + f_p(x_{ip}) + \epsilon_i$$. A separate non-linear function $$f_j$$ is applied to every predictor variable, and these are then added together. This is why it is considered an additive model.

GAMs in Regression Problems

Similar to GAMs in regression problems, GAMs can also be used when responses are qualitative. Assuming that Y takes the value of either 0 or 1, we can start off with the logistic regression model $log( \frac{p(X)}{1-p(X)}) = \beta _0 +  \beta _1X_{1} +  \beta _2X_{2}+...+ \beta _pX_{p}$ and replace the linear component ($\beta_jX_j$) with separate non-linear functions ($f_j(X_j)$) for every predictor variable. Our corresponding model is thus $log( \frac{p(X)}{1-p(X)}) = \beta _0 +  f_1(X_{1}) +  f_2(X_{2})+...+ f_p(X_{p})$.

GAMs for Classification Problems

https://codeburst.io/pygam-getting-started-with-generalized-additive-models-in-python-457df5b4705f



GAM implementation in Python

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value ($$Y$$) using many predictor variables ($$X_1, \dots, X_p$$). GAMs maintain their additivity by applying separate non-linear functions to each predictor variable ($$f_j(X_j)$$) and adding them together. They can be applied to both quantitative and qualitative responses.

University of Michigan - Ann Arbor

Google

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Local regression works by choosing a target point ($$x_0$$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $$K_{i0} = K(x_i, x_0)$$) based on their distance from $$x_0$$; points closest to $$x_0$$ are assigned the highest weight, and as points get farther from $$x_0$$, they are assigned a lower weight. The point farthest from $$x_0$$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing $$\sum_{i=1}^{n} K_{i0} (y_i - \beta_0- \beta_1x_i)^{2}$$. The new fitted value at $$x_0$$ is then found using the new regression model.

Local Regression

In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


Smoothing Splines

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

Regression splines divide the range of independent variables ($$X$$) into $$K$$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75. ![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

Regression Splines

Generalized Additive Models (GAMs)

The basis function approach to regression involves using a family of functions or transformations that can be applied to a variable $$X$$: $$b_1(X), b_2(X), \cdots, b_K(X)$$. Instead of fitting a standard linear model in $$X$$, we fit the model: $$y_i = \beta_0 + \beta_1b_1(x_i) + \beta_2b_2(x_i) + \cdots + \beta_Kb_K(x_i) + \epsilon_i$$. Note that the basis functions $$b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$$ are fixed and known, meaning they are chosen ahead of time.

Basis Functions for Regression

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Artificial Neural Networks are built to resemble neurons in the human brain. The idea behind neural networks is to understand the pattern behind terms through examples rather than through task-specific rules.

Artificial Neural Networks

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

Learn Before

Related

Learn After