If we simply ﬁt a smooth curve $g(x)$ to a set of data by minimizing RSS = $\sum^n_{i=1}(y_i - g(x_i))^2$ without putting any constraints, there is a problem that we can always make RSS zero simply by choosing $g$ such that it interpolates all of the $y_i$, leading to the overfitting of the data. What we want is a function $g$ that makes RSS small, but that is also smooth.
Instead, we can change the optimization formulation from RSS to the function below:
$$\sum^n_{i=1}(y_i - g(x_i))^2 + \lambda \int g''(t)^2 dt$$,
where λ is a nonnegative tuning parameter. The function $g$ that minimizes the formulation above is known as a smoothing spline.

Smoothing splines avoid overfitting

Regression splines combine the benefits of polynomial curve fitting and step functions to provide a more flexible estimated function. By dividing the range of the independent variable $$X$$ into enough regions, we can theoretically fit any function. However, this flexibility comes at the expense of potential overfitting and reduced interpretability of the coefficients. Therefore, a trade-off must be considered when deciding on the number of regions.

University of Michigan - Ann Arbor

Google

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

Regression splines divide the range of independent variables ($$X$$) into $$K$$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75. ![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

Regression Splines

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Step functions are commonly used in epidemiology and biostatistics for accounting for biological changes due to factor like age (age binning)

Common uses cases of step functions.

Regression Splines vs. Polynomial Curve Fitting and Step Functions

Estimating the coefficients of a polynomial regression is not very different from estimating the coefficients in linear regression. The only difference is that instead of completely independent predictors, a predictor can be exponentiation of another predictor. So, we can still use least squares to estimate them.

Estimate in a polynomial regression

```python
#imports 
import numpy as np 
from sklearn.linear_model import LinearRegression 
from sklearn.preprocessing import PolynomialFeatures

#creating sample data 
array_x = np.array([5, 10, 15, 20, 25]).reshape((-1, 1))
~ calling reshape makes sure that the array is 2D 
array_y = np.array([5, 3, 15, 23, 6])

#transforming data 
transformed = PolynomialFeatures(degree = 2, include_bias = False)
#degree = 2, so this is a binomial regression 
#then fit it 
transformed.fit(array_x)

#now it has been fitted, so let’s create a new input 
x_mod = transformed.transform(array_x)

#create a model and fit it 
model = LinearRegression().fit(x_mod, y)

#get results 
r_squared = model.score(x_mod, y)
intercept = model.intercept_
coefficients = model.coef_

#predict response 
y_pred = model.predict(x_)
```

Python: Polynomial Regression

A special type of polynomial regression where the $$X$$, $$X^2$$, and $$X^3$$ are used as predictors (independent variables). $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \epsilon_i$$

Cubic regression

Regression splines often give superior results compared to polynomial regression. Unlike polynomials, which must use a high degree (the exponent in the highest monomial term, e.g., $$X^{15}$$) to produce flexible fits, splines introduce flexibility by increasing the number of knots but keeping the degree fixed. As shown in the accompanying figure, polynomials can exhibit unexpected and wild behavior, especially near the tails. This is not preferred for larger range predictions.

Regression Splines vs. Polynomial Regression

https://data.princeton.edu/eco572/smoothing2

Example of Regression Splines

While it may seem intuitive to place more knots where the data has more variance, in practice it is best to uniformly distribute the knots in the data as to not overfit in certain areas. Commonly the number and placement of knots is determined by selecting your desired degrees of freedom and letting data software distribute them automatically. 

Learn Before

Related

Learn After