Regression splines combine the benefits of polynomials and step functions to provide us with a more flexible estimated function. Assuming that we divide the range of independent variables $X$ into enough regions, we can theoretically fit any function. However, this flexibility comes with the expense of overfitting and less interpretability of the coefficients. So, the analyzer should consider a trade-off when deciding on the number of regions.

Regression splines vs. polynomials and step functions

Step functions are commonly used in epidemiology and biostatistics for accounting for biological changes due to factor like age (age binning)

Common uses cases of step functions.

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

University of Michigan - Ann Arbor

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Local regression works by choosing a target point ($x_0$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $K_{i0} = K(x_i, x_0)$) based on their distance from $x_0$; points closest to $x_0$ are assigned the highest weight, and as points get farther from $x_0$, they are assigned a lower weight. The point farthest from $x_0$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing  $$\sum_{i=1}^{n} K_{i0} (y_i -  \beta_0- \beta_1x_i)^{2}$$. The new fitted value of $x_0$ is then found using the new regression model.

Local Regression

In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


Smoothing Splines

Regression splines divide the range of independent variables ($X$) into $K$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75.
![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

Regression Splines

 The idea is to have  a family of functions or transformations that can be applied to a variable $X$: $b_1(X), b_2(X),\cdots, b_K(X)$. Instead of fitting the linear model, we fit the model: 
 $y_i = \beta_0+\beta_1b_1(x_i)+\beta_2b_2(x_i)+\cdots+\beta_Kb_K(x_i)+\epsilon_i$
 
 Note that $b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$ are fixed and known (chosen ahead of time). 
 

Basis Functions

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Learn Before

Related

Learn After