Estimating the coefficients of a polynomial regression is not very different from estimating the coefficients in linear regression. The only difference is that instead of completely independent predictors, a predictor can be exponentiation of another predictor. So, we can still use least squares to estimate them.

University of Michigan - Ann Arbor

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

A special type of polynomial regression where the $X$, $X^2$, and $X^3$ are used as predictors (independent variables).
$$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i ^3 + \epsilon_i$$ 

Cubic regression

Regression splines combine the benefits of polynomials and step functions to provide us with a more flexible estimated function. Assuming that we divide the range of independent variables $X$ into enough regions, we can theoretically fit any function. However, this flexibility comes with the expense of overfitting and less interpretability of the coefficients. So, the analyzer should consider a trade-off when deciding on the number of regions.

Regression splines vs. polynomials and step functions

Regression splines often give superior results to polynomial regression. This is because unlike polynomials, which must use a high degree (exponent in the highest monomial term, e.g. $X^{15}$) to produce ﬂexible ﬁts, splines introduce ﬂexibility by increasing the number of knots but keeping the degree ﬁxed. 
From the figure below, we can see that polynomials can show unexpected and wild behavior, especailly near the tails. This is not preferred in the larger range prediction.



Regression splines vs. polynomial regression

Estimate in a polynomial regression

```python
#imports 
import numpy as np 
from sklearn.linear_model import LinearRegression 
from sklearn.preprocessing import PolynomialFeatures

#creating sample data 
array_x = np.array([5, 10, 15, 20, 25]).reshape((-1, 1))
~ calling reshape makes sure that the array is 2D 
array_y = np.array([5, 3, 15, 23, 6])

#transforming data 
transformed = PolynomialFeatures(degree = 2, include_bias = False)
#degree = 2, so this is a binomial regression 
#then fit it 
transformed.fit(array_x)

#now it has been fitted, so let’s create a new input 
x_mod = transformed.transform(array_x)

#create a model and fit it 
model = LinearRegression().fit(x_mod, y)

#get results 
r_squared = model.score(x_mod, y)
intercept = model.intercept_
coefficients = model.coef_

#predict response 
y_pred = model.predict(x_)
```

Learn Before

Related