Step functions are commonly used in epidemiology and biostatistics for accounting for biological changes due to factor like age (age binning)

Common uses cases of step functions.

Regression splines combine the benefits of polynomial curve fitting and step functions to provide a more flexible estimated function. By dividing the range of the independent variable $$X$$ into enough regions, we can theoretically fit any function. However, this flexibility comes at the expense of potential overfitting and reduced interpretability of the coefficients. Therefore, a trade-off must be considered when deciding on the number of regions.

Regression Splines vs. Polynomial Curve Fitting and Step Functions

A step-function model replaces one global linear fit with several flat pieces. First, split the predictor range into intervals, or bins. Then estimate a separate constant response level within each bin.

Because the fitted value changes only when the predictor crosses a bin boundary, the model is easy to interpret: each interval has its own average outcome. This makes a continuous variable behave like an ordered categorical variable, which can be helpful when the response changes in distinct ranges rather than smoothly.

University of Michigan - Ann Arbor

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Local regression works by choosing a target point ($$x_0$$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $$K_{i0} = K(x_i, x_0)$$) based on their distance from $$x_0$$; points closest to $$x_0$$ are assigned the highest weight, and as points get farther from $$x_0$$, they are assigned a lower weight. The point farthest from $$x_0$$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing $$\sum_{i=1}^{n} K_{i0} (y_i - \beta_0- \beta_1x_i)^{2}$$. The new fitted value at $$x_0$$ is then found using the new regression model.

Local Regression

In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


Smoothing Splines

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

Regression splines model a nonlinear relationship by splitting the range of a predictor variable X into K intervals and fitting a separate polynomial on each interval. The pieces are estimated so that the final curve joins smoothly at the knot locations. For example, if X runs from 0 to 100 and knots are placed at 25, 50, and 75, one polynomial is fit on 0-25, another on 25-50, and so on, with the pieces connected at each knot.

How Regression Splines Work

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value ($$Y$$) using many predictor variables ($$X_1, \dots, X_p$$). GAMs maintain their additivity by applying separate non-linear functions to each predictor variable ($$f_j(X_j)$$) and adding them together. They can be applied to both quantitative and qualitative responses.

Generalized Additive Models (GAMs)

The basis function approach to regression involves using a family of functions or transformations that can be applied to a variable $$X$$: $$b_1(X), b_2(X), \cdots, b_K(X)$$. Instead of fitting a standard linear model in $$X$$, we fit the model: $$y_i = \beta_0 + \beta_1b_1(x_i) + \beta_2b_2(x_i) + \cdots + \beta_Kb_K(x_i) + \epsilon_i$$. Note that the basis functions $$b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$$ are fixed and known, meaning they are chosen ahead of time.

Learn Before

Related

Learn After