In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


University of Michigan - Ann Arbor

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

The no free lunch theorem is proof that no single regularization is strategy is certifiably the best at every task, however that is not to say that every strategy is created equal. There may be no best general strategy, but there may be strategies that are often times more useful than others. That being the case, here is a list of some regularization assumptions/ strategies that have been shown to be generally useful:

- Smoothness
- Linearity
- Multiple explanatory factors
- Causal factors
- Depth/ hierarchical factors
- Shared factors across tasks
- Manifolds
- Natural clustering
- Temporal and spatial coherence
- Sparsity
- Simplicity of factor dependencies.


Appropriate Regularization/ Representation

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Local regression works by choosing a target point ($x_0$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $K_{i0} = K(x_i, x_0)$) based on their distance from $x_0$; points closest to $x_0$ are assigned the highest weight, and as points get farther from $x_0$, they are assigned a lower weight. The point farthest from $x_0$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing  $$\sum_{i=1}^{n} K_{i0} (y_i -  \beta_0- \beta_1x_i)^{2}$$. The new fitted value of $x_0$ is then found using the new regression model.

Local Regression

Smoothing Splines

Regression splines divide the range of independent variables ($X$) into $K$ distinct chunks and fit a polynomial regression to each chunk to eventually form one model. The estimation of the polynomials is constrained in the predetermined ranges, so they connect smoothly together. Below the data is fit in chunks, with the separately fit chunks meeting at 25, 50, and 75.
![Image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FCharlie-Logan_Fri%2C%2021%20Feb%202020%2020%3A10%3A23%20GMT.png?alt=media&token=856ee94d-fbe5-4199-bbb0-5810ade2c241)

Regression Splines

 The idea is to have  a family of functions or transformations that can be applied to a variable $X$: $b_1(X), b_2(X),\cdots, b_K(X)$. Instead of fitting the linear model, we fit the model: 
 $y_i = \beta_0+\beta_1b_1(x_i)+\beta_2b_2(x_i)+\cdots+\beta_Kb_K(x_i)+\epsilon_i$
 
 Note that $b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$ are fixed and known (chosen ahead of time). 
 

Basis Functions

Instead of using a linear model to fit the data gloabally, we can fit the data into a model of step functions. As is shown below,  Here we break the range of X into bins, and ﬁt a diﬀerent constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable.
![Image from the cited textbook](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Fri%2C%2021%20Feb%202020%2018%3A06%3A02%20GMT.png?alt=media&token=7480fe96-a295-4a49-888c-f281b6946c7d)

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

In a Feed-forward NN, we use the linear matrix multiplication for the forward propagation and each element in the output is dependent on each element in the input. Now imagine the input is a 500 x 500 image - including the three image channels, we have the input of 3 x 500 x 500 = 750,000 dimensions. Considering this number as only one side of the parameter matrix in the first layer, this means a lot of parameters in the first layer.

In contrast, in a convolution layer, each output value only depends on a small number of inputs, allowing us to significantly improve on efficiency, and decrease memory requirements. This way, we will be able to use very large input images.

Sparsity of Connections in Convolutional Neural Networks

Causal inference refers to making conclusions about finding the reasonings behind the relationship of two or more variables. After coming to some conclusions about statistical data, we ask why such relationships exist between variables.

Causal Inference

Clustering tries to place unlabeled samples into groups based on similar characteristics. 
An example, these samples can be distinguished into 5 categories by some observed characteristic. Let's say we are clustering textbook samples by different school subjects (math, science, history, english, art). 
![The image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FMadeline-Rosenberg_Wed%2C%2019%20Feb%202020%2015%3A27%3A54%20GMT.png?alt=media&token=5b890a1d-803b-4ad5-a65f-1ec4043225c2) 

Clustering, an unsupervised statistical learning method

 Good at finding low dimensional structure in high dimensional data 
Methods:
Multidimensional scaling (MDS): attempts to find a distance-preserving low-dimensional projection
Preserves information about how points in original dataspace are close to each other


Manifold learning algorithms

A feedforward network with a single layer is suﬃcient to representany function, but the layer may be infeasibly large and may fail to learn and generalize correctly. In many circumstances, using deeper models can reduce the number of units required to represent the desired function and can reduce the amount of generalization error.

As is shown in the figure, empirical results show that deeper networks generalize better when used to transcribe multidigit numbers from photographs of addresses. Data from Goodfellow et al. (2014d). The test set accuracy consistently increases with increasing depth.


Effect of Depth for Neural Networks

While a parameter norm penalty is one way to regularize parameters to be close to one another, the more popular way is to use constraints: to force sets of parameters to be equal. This method of regularization is often referred to as parameter sharing, because we interpret the various models or model components as sharing a unique set of parameters. A signiﬁcant advantage of parameter sharing over regularizing the parameters to be close (via a norm penalty) is that only a subset of the parameters (the unique set) needs to be stored in memory. In certain models—such as the convolutional neural network—this can lead to signiﬁcant reduction in the memory footprint of the model.



	Parameter sharing: to force sets of parameters to be equal

	Main application: Convolutional Neural Network (CNN)

	Advantages: signiﬁcant reduction in the memory footprint of the model




Parameter Sharing

The idea that the exact value of any underlying factors that created a given input may change over time or distance. That being the case, any difference in time or distance between when the input was captured compared to the training output may result in some noise corresponding to the exact space-time discrepancy. As such, it may be that the underlying factors that caused the input are easier to determine than the precise output value that was recorded.

Temporal and Spatial Coherence

The assumption that the relationships between underlying factors are simple to express. As an example, it is common to assume that these factors are independent of each other. This results in the probability of any factor set $h$ is simply equal to the product of the probabilities for each  factor state $h_i$.
$$P(h) = \prod P(h_i)$$
Many machine learning algorithms assume either this independence or some other simple/ easy to calculate relationship.

Learn Before

Related