If we simply ﬁt a smooth curve $g(x)$ to a set of data by minimizing RSS = $\sum^n_{i=1}(y_i - g(x_i))^2$ without putting any constraints, there is a problem that we can always make RSS zero simply by choosing $g$ such that it interpolates all of the $y_i$, leading to the overfitting of the data. What we want is a function $g$ that makes RSS small, but that is also smooth.
Instead, we can change the optimization formulation from RSS to the function below:
$$\sum^n_{i=1}(y_i - g(x_i))^2 + \lambda \int g''(t)^2 dt$$,
where λ is a nonnegative tuning parameter. The function $g$ that minimizes the formulation above is known as a smoothing spline.

Smoothing splines avoid overfitting

Regression splines combine the benefits of polynomials and step functions to provide us with a more flexible estimated function. Assuming that we divide the range of independent variables $X$ into enough regions, we can theoretically fit any function. However, this flexibility comes with the expense of overfitting and less interpretability of the coefficients. So, the analyzer should consider a trade-off when deciding on the number of regions.

University of Michigan - Ann Arbor

Regression splines divide the range of independent variables ($X$) into $K$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75.
![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

Regression Splines

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

https://data.princeton.edu/eco572/smoothing2

Example of Regression Splines

Regression splines vs. polynomials and step functions

Regression splines often give superior results to polynomial regression. This is because unlike polynomials, which must use a high degree (exponent in the highest monomial term, e.g. $X^{15}$) to produce ﬂexible ﬁts, splines introduce ﬂexibility by increasing the number of knots but keeping the degree ﬁxed. 
From the figure below, we can see that polynomials can show unexpected and wild behavior, especailly near the tails. This is not preferred in the larger range prediction.



Regression splines vs. polynomial regression

While it may seem intuitive to place more knots where the data has more variance, in practice it is best to uniformly distribute the knots in the data as to not overfit in certain areas. Commonly the number and placement of knots is determined by selecting your desired degrees of freedom and letting data software distribute them automatically. 

Regression Splines: Deciding how many knots to use and where to place them

Step functions are commonly used in epidemiology and biostatistics for accounting for biological changes due to factor like age (age binning)

Common uses cases of step functions.

A special type of polynomial regression where the $X$, $X^2$, and $X^3$ are used as predictors (independent variables).
$$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i ^3 + \epsilon_i$$ 

Cubic regression

Estimating the coefficients of a polynomial regression is not very different from estimating the coefficients in linear regression. The only difference is that instead of completely independent predictors, a predictor can be exponentiation of another predictor. So, we can still use least squares to estimate them.

Estimate in a polynomial regression

```python
#imports 
import numpy as np 
from sklearn.linear_model import LinearRegression 
from sklearn.preprocessing import PolynomialFeatures

#creating sample data 
array_x = np.array([5, 10, 15, 20, 25]).reshape((-1, 1))
~ calling reshape makes sure that the array is 2D 
array_y = np.array([5, 3, 15, 23, 6])

#transforming data 
transformed = PolynomialFeatures(degree = 2, include_bias = False)
#degree = 2, so this is a binomial regression 
#then fit it 
transformed.fit(array_x)

#now it has been fitted, so let’s create a new input 
x_mod = transformed.transform(array_x)

#create a model and fit it 
model = LinearRegression().fit(x_mod, y)

#get results 
r_squared = model.score(x_mod, y)
intercept = model.intercept_
coefficients = model.coef_

#predict response 
y_pred = model.predict(x_)
```

Learn Before

Related

Learn After