https://data.princeton.edu/eco572/smoothing2

Example of Regression Splines

Regression splines combine the benefits of polynomials and step functions to provide us with a more flexible estimated function. Assuming that we divide the range of independent variables $X$ into enough regions, we can theoretically fit any function. However, this flexibility comes with the expense of overfitting and less interpretability of the coefficients. So, the analyzer should consider a trade-off when deciding on the number of regions.

Regression splines vs. polynomials and step functions

Regression splines often give superior results to polynomial regression. This is because unlike polynomials, which must use a high degree (exponent in the highest monomial term, e.g. $X^{15}$) to produce ﬂexible ﬁts, splines introduce ﬂexibility by increasing the number of knots but keeping the degree ﬁxed. 
From the figure below, we can see that polynomials can show unexpected and wild behavior, especailly near the tails. This is not preferred in the larger range prediction.



Regression splines vs. polynomial regression

While it may seem intuitive to place more knots where the data has more variance, in practice it is best to uniformly distribute the knots in the data as to not overfit in certain areas. Commonly the number and placement of knots is determined by selecting your desired degrees of freedom and letting data software distribute them automatically. 

Regression Splines: Deciding how many knots to use and where to place them

Regression splines divide the range of independent variables ($X$) into $K$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75.
![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

University of Michigan - Ann Arbor

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Local regression works by choosing a target point ($x_0$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $K_{i0} = K(x_i, x_0)$) based on their distance from $x_0$; points closest to $x_0$ are assigned the highest weight, and as points get farther from $x_0$, they are assigned a lower weight. The point farthest from $x_0$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing  $$\sum_{i=1}^{n} K_{i0} (y_i -  \beta_0- \beta_1x_i)^{2}$$. The new fitted value of $x_0$ is then found using the new regression model.

Local Regression

In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


Smoothing Splines

Regression Splines

 The idea is to have  a family of functions or transformations that can be applied to a variable $X$: $b_1(X), b_2(X),\cdots, b_K(X)$. Instead of fitting the linear model, we fit the model: 
 $y_i = \beta_0+\beta_1b_1(x_i)+\beta_2b_2(x_i)+\cdots+\beta_Kb_K(x_i)+\epsilon_i$
 
 Note that $b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$ are fixed and known (chosen ahead of time). 
 

Basis Functions

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Learn Before

Related

Learn After