Getting more data may be costly and an ineffective way to prevent overfitting. However, you can augment your data to create more. We are adding synthetic data modified from our original data set. You essentially transform your data in a way where it is different form the original, but the data still fits into your given class. It is much more efficient and cost effective. 

Data Augmentation in Deep Learning

Early stopping is a classic regularization technique for deep neural networks that mitigates overfitting by constraining the number of training epochs instead of directly penalizing weight values. This approach is motivated by the fact that neural networks tend to fit clean data before memorizing noisy labels; by halting training at the optimal epoch, the model avoids interpolating noise and thereby improves generalization.

Early Stopping in Deep Learning

Unlike L2 or L1 regularization, Dropout regularization works by assigning a probability P, which is the probability that a given node is turned off for the current iteration of training. By doing this, you make the model simpler while training, which will reduce overfitting, while still gaining the benefits a larger model has during testing. Also, no single node will have excess influence on the model, and the model will spread the weights out, giving us a similar effect to L2 regularization. Generally, Dropout Regularization is the preferred form of regularization, as it not only accomplishes results similar to L2 regularization, it also provides some robustness to the model as each iteration is randomized.

Dropout Regularization in Deep Learning

Which of these techniques are useful for reducing variance (reducing overfitting)?

In statistics and, in particular, in the fitting of linear or logistic regression models, the elastic net is a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods.

ElasticNet Regression

If your Neural Network model seems to have high variance, what of the following would be promising things to try?

https://medium.com/analytics-vidhya/regularization-in-machine-learning-and-deep-learning-f5fa06a3e58a

Regularization in ML and DL

- Bagging is short for bootstrap aggregating.

- Bagging is a technique for reducing generalization error by combining several machine learning models.

- Bagging employs model averaging.

- This is powerful method of regularization, which is widely used in machine learning contest. However, it is quite impractical, since the computational cost of training several models in expensive.
 


Bagging in Deep Learning

- Instead of training multiple models like bagging, dropout train the ensemble consisting of all subnetworks that can be formed by removing non-output units from an underlying base network. These models share parameters, with each model inheriting a different subset of parameters from parent networks. 

- We need to specify the probability of unit to be included. Typically, the input unit is included with probability 0.8 while that in hidden units is 0.5.

-  Prediction of ensemble is given by geometric mean.


- Cheap computational cost and can be combined with other method or regularization.



Dropout in Deep Learning 

Normalization is one of the techniques used to reduce the bias and variance in statistical learning. It helps to have all the variables centered around 0 and have the same variance.
We subtract all the values of each variable from their mean and divide them by their variance.
$$\mu = \frac{1}{m} \sum_{i = 1}^{m} X^{(i)}$$
$$\sigma^{2} = \frac{1}{m} \sum_{i = 1}^{m} (X^{(i)} - \mu)^2$$
$$X_{New} = \frac{X - \mu}{\sigma}$$

Normalization of Data

- Early attempt to take advantage of the manifold hypothesis
- Nonparametric nearest neighbor algorithm, metric used is derived from the manifolds near which probability concentrates
- Assumes that data on the same manifold all has the same category
- Classifier should be invariant to local factors of variation that correspond to movement. So we use the nearest neighbor distance between two points, which is the distance between the manifolds they belong to
- Cheap alternative on a local level is to approximate each manifold by its tangent plane at a point and measure the distance between the two tangents, or between the tangent plane and a point, by solving a low-dimensional linear system.

Tangent Distance Algorithm

- Similar to tangent distance algorithm
- Closely related to dataset augmentation, both require that the model be invariant to certain specified directions of change in the input. Dataset augmentation is the non-infinitesimal version of tangent propagation
- Trains a neural net classifier with extra penalty to make each output of the neural net locally invariant to known factors of variation
- Factors correspond to movement along the manifold near which examples of the same class concentrate.
- Local invariance achieved by requiring $\triangledown_x  f(x)$  to be orthogonal to known manifold tangent vectors $v^{(i)}$ at $x$
- Equivalently, the directional derivative of $f$ at $x$ in the directions $v^{(i)}$ be small by adding a regularization penalty $\omega$, defined as: $\Omega(f) = \sum_{i} ((\triangledown_x f(x))^\top v^{(i)})^2$, which can be scaled by a hyperparameter, and for most neural networks, we would need to sum over many outputs
- Tangent vectors are derived a priori, usually from knowledge of the effect of transformations
- Has been used for supervised learning and reinforcement learning
- User encodes prior knowledge of task by specifying a set of transformations that should not alter the output, and analytically regularizes the model to resist perturbation in the directions corresponding to the specified transformation
- Only regularizes the model to resist infinitesimal perturbation, and poses difficulties for models based on rectified linear units
- Related to double backprop and adversarial training, both of which require that the model should be invariant to all directions of change in the input as long as the change is small
- Double backprop regularizes the Jacobian to be small
- Adversarial training finds inputs near the original inputs and trains the model to produce the same output on these as on the original inputs
- Adversarial training is the non-infinitesimal version of double backprop


Tangent Propagation Algorithm

Eliminates the need to know the tangent vectors a priori. Uses autoencoders to estimate manifold tangent vectors to avoid needing user-specified tangent vectors, which go beyond the classical invariants from the geometry of images and include factors that must be learned because they are object-specific. The algorithm first uses an autoencoder to learn the manifold structure by unsupervised learning, then uses the tangents to regularize a neural net classifier, similar to tangent propagation.

Manifold Tangent Classifier

- Boosting helps to build a strong ensemble( ensemble is a collection of multiple machine learning models) compared to a good capacity( strong learning model) individual machine learning model.

- In the context of deep learning all machine learning models are neural networks.

- Individual neural network is also considered as an ensemble which is improved by incrementally adding hidden layers.

- Output from the ensemble is combined to predict the output.

Boosting in Deep Learning

The no free lunch theorem is proof that no single regularization is strategy is certifiably the best at every task, however that is not to say that every strategy is created equal. There may be no best general strategy, but there may be strategies that are often times more useful than others. That being the case, here is a list of some regularization assumptions/ strategies that have been shown to be generally useful:

- Smoothness
- Linearity
- Multiple explanatory factors
- Causal factors
- Depth/ hierarchical factors
- Shared factors across tasks
- Manifolds
- Natural clustering
- Temporal and spatial coherence
- Sparsity
- Simplicity of factor dependencies.


Appropriate Regularization/ Representation

Weight decay, commonly known as $$\ell_2$$ regularization, is a widely used technique for regularizing parametric machine learning models. Instead of directly manipulating the number of parameters, weight decay operates by restricting the values that the parameters can take. The technique is motivated by the intuition that the simplest function is $$f = 0$$, and the complexity of a linear function, such as $$f(\mathbf{x}) = \mathbf{w}^	op \mathbf{x}$$, can be measured by the distance of its parameters from zero.

Weight Decay

Unlike $$\ell_2$$ regularization which distributes weights evenly, $$\ell_1$$ regularization penalizes the absolute values of the weights. This leads to models that concentrate weights on a small set of features by clearing the other weights to zero, making it an effective method for feature selection. If a model only relies on a few features, it may eliminate the need to collect, store, or transmit data for the dropped features. Linear models that are $$\ell_1$$-regularized are popularly known as lasso regression.

L1 Regularization

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

University of Michigan - Ann Arbor

University of Texas at San Antonio

Linear regression is a linear approach to model a target prediction value based on independent variables.

Linear Regression

Regularization is a modification made to a learning algorithm with the goal of reducing generalization error, but not training error.

Linear Regression is useful for modeling outputs whose relation to their inputs is linear. However, linear regression performs poorly when trying to model non-linear relationships. We can use regularization to help learning algorithms select certain models over others, particularly to avoid overfitting. 

Regularization

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Based on the number of independent variables used, there are two types of linear regression:
- Simple Linear Regression
- Multiple Linear Regression

Types of linear regression

Logistic regression is used to model discontinuous output choices - for example, when a variable is 'yes' or 'no' rather than a continuous value. In this case, logistic regression calculates the probability of a given categorical output, and decides on a threshold at which the model decides between categories to select for output. 

Logistic Regression

The terms of a linear model can be one of the following:
	
	- The Constant
	- A parameter multiplied by an independent variable (I.V)


Then, you build the equation by only adding the terms together. These rules limit the form to just one type:
Dependent variable = constant + parameter * IV + … + parameter * IV

Nonlinear regression models don’t fit this type.

Examples of nonlinear models include: 
	

When is a model considered linear?

This is a series of videos that introduces linear regression models and breaks down the process of linear regression.

[StatQuest with Josh Starmer]. (2020, February 21). Linear Regression and Linear Models [Video File]. Retrieved from [www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU](https://www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU) 

Linear Regression and Linear Models Videos

Here is an example of using R to do a simple linear regression. 
```R
#MASS package is a very large collection of data sets and #functions
library(MASS) 

#ISLR package includes the data sets associated with this #book
library(ISLR) 

lm.fit=lm(medv∼lstat,data=Boston) 
# medv as a response, lstat as a predictor

summary(lm.fit) # gives us pvalues and standard errors for the coeﬃcients, as well as the R2 statistic and F-statistic for the model

confint(lm.fit) # obtain a confidence interval

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="confidence") #prediction in confidence intervals

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="prediction") #prediction in prediction intervals

attach(Boston) # attach data to Boston

plot(lstat ,medv) 
abline(lm.fit ,lwd=3,col="red") 
# draw out the regression line
```
Expected output (plot and abline only):
![The expected output from the plot and abline commands](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Wed%2C%2026%20Feb%202020%2020%3A51%3A58%20GMT.png?alt=media&token=5cf41866-c6fe-4732-88c5-2233fe889aa3) 

Example of Linear Regression Using R

We estimate the parameters $\beta_0, \beta_1, \beta_2, \dots, \beta_p$ such that we minimize the error in the following approximation:
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

Training or Fitting a linear regression model

The extent to which the linear regression model fits the data, the quality of linear regression fit, can be achieved by:
1. $ R^2 $
2. Residual Standard Error
3. Goodnesss of Fit
4. F Statistics
5. Mean Square Error (MSE)

Assessing the accuracy of linear regression

Linear regression is an example of a parametric method because it assumes a linear functional form for f(X). But if the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. This leads to higher bias. There is also a well known statistical theory behind linear regression.

KNN is an example of non parametric method that uses feature similarity. It is pretty useful, because in the real world, most of the practical data does not obey the typical theoretical assumptions made (eg gaussian mixtures, linearly separable etc). Non-parametric methods do not explicitly assume a parametric form for f(X), and thereby provide an alternative and more flexible approach for performing regression. This means it will have less bias and is more robust with highly non-linear settings. However, KNN regression is not as widely studied as linear regression.

Comparison of Linear Regression with K-Nearest Neighbors

- Linearity of the response-predictor relations
- Normal distribution of residuals
-  Error terms, e1,e2,...e3, are uncorrelated
-  Error terms have a constant variance
-  Little or no Multicollinearity between the features
-  Homoscedasticity Assumption
-  Large sample size and few or no outliers

Assumptions of Linear Regression

The regression coefficient is $$\beta_1$$, in the regression equation $$y \approx \beta_0 + \beta_1 X$$, is an esitmate of the rate of change (slope) between X and Y.

Regression Coefficient

It is easier to implement, interpret, and infer from the estimation of linear regression. However, one can rarely find a perfect linear relationship in the real world. Estimation using nonlinear regression is more difficult to implement, interpret, and infer from, but it gives us more predictive power by relaxing the assumption of linearity of the model.

Linear regression vs. nonlinear regression

Popular Regularization Techniques in Deep Learning

We can use  polynomial future transformation to transform a problem into a higher dimensional regression space. Adding these extra polynomial features allows us a much richer set of complex functions that we can use to fit to the data. This is like allowing polynomials to be fit to the training data instead of simply a straight line, but using the same least-squares criterion that minimizes mean squared error. This approach of adding new features like polynomial features is also very effective with classification.

Polynomial Future Transformation

To make up for the under-fitting or over-fitting problems of ordinary linear regression models, locally weighted linear regression introduce weights to loss function. 
Its loss is transmitted to:
$J(\theta)=\sum_{i=1}^m w^i (y^{(i)} - \theta^T x^{(i)})^2$
w is used to represent the weights. Its value is calculated using Gaussian kernel function:
$w^i = exp(\frac{(x-x^i)^2}{-2k^2})$

Locally Weighted Linear Regression

In the context of linear regression, specific mathematical notation is used to systematically describe the dataset. Typically, $$n$$ denotes the total number of examples. Superscripts are utilized to enumerate individual samples and their corresponding targets, while subscripts are used to index their specific coordinates or features. For instance, $$\mathbf{x}^{(i)}$$ represents the complete $$i^{\textrm{th}}$$ sample, and $$x_j^{(i)}$$ designates the $$j^{\textrm{th}}$$ coordinate of that $$i^{\textrm{th}}$$ sample.

Linear Regression Dataset Notation

The foundational concepts of linear regression date back to the early 19th century, with significant contributions from mathematicians Carl Friedrich Gauss in 1809 and Adrien-Marie Legendre in 1805. Despite its historical age, linear regression remains one of the simplest and most popular standard tools for tackling modern regression problems.

History of Linear Regression

In the simplest case of one-dimensional inputs, a linear regression model attempts to fit a straight line through the observed data points. The fitness of this line depends purely on the choice of model parameters (the weight and the bias), which are adjusted to minimize the distance between the line's predictions and the true target values.

Linear Regression One-Dimensional Fit

The squared error is the standard loss function for regression problems, used to quantify the discrepancy between a single predicted value $$\hat{y}^{(i)}$$ and its true corresponding label $$y^{(i)}$$. The loss for a single example is defined mathematically as $$l^{(i)}(\mathbf{w}, b) = \frac{1}{2} \left(\hat{y}^{(i)} - y^{(i)}ight)^2$$. The quadratic form heavily penalizes large differences, and the constant fraction $$\frac{1}{2}$$ is conventionally added because it cleanly cancels out when taking the derivative during optimization.

Squared Error Loss

A fundamental assumption of linear regression is that the relationship between the features $$\mathbf{x}$$ and the target $$y$$ is approximately linear. This means the expected value of the target, defined as the conditional mean $$E[Y \mid X=\mathbf{x}]$$, can be mathematically expressed as a weighted sum of the input features $$\mathbf{x}$$. This framework allows the actual target values to deviate from their expected conditional mean due to observation noise.

Linear Regression Conditional Mean Assumption

In a multiple-feature linear regression model, weights are parameters that determine the influence of each individual feature on the target prediction. When the model is expressed as a weighted sum, such as $$\hat{y} = w_1 x_1 + \dots + w_d x_d + b$$, each weight quantifies the specific contribution of its corresponding feature. In compact linear algebra notation, these parameters are collected into a weight vector $$\mathbf{w} \in \mathbb{R}^d$$.

Linear Regression Weight Parameters

In a linear regression model, the bias (also referred to as an offset or intercept) is a parameter that determines the baseline value of the model's estimate when all input features are exactly zero. The addition of the bias term $$b$$ is mathematically necessary because it permits the model to express general linear functions as an affine transformation, rather than restricting predictions to lines that must pass directly through the origin.

Linear Regression Bias Parameter

Linear regression can be conceptualized as a single-layer, fully connected neural network. In this representation, each given input feature $$x_1, \ldots, x_d$$ corresponds to an input neuron. Since the goal is to predict a single numerical value, all input neurons are directly connected to a single computed output neuron $$o_1$$. The total number of inputs $$d$$ is referred to as the feature dimensionality in the input layer.

Linear Regression as a Neural Network

To evaluate machine learning models, we often generate synthetic datasets where the underlying ground truth relationship is known. For a linear regression task, we can draw a design matrix $$\mathbf{X}$$ of features from a standard normal distribution. The corresponding labels $$\mathbf{y}$$ are computed by applying a ground truth linear function defined by true weights $$\mathbf{w}$$ and bias $$b$$, and then corrupting the output with additive noise $$\boldsymbol{\epsilon}$$ drawn from a normal distribution with mean $$\mu=0$$ and standard deviation $$\sigma = 0.01$$: $$\mathbf{y}= \mathbf{X} \mathbf{w} + b + \boldsymbol{\epsilon}$$. This procedure ensures that the generated labels simulate realistically observed data containing inherent random variation.

Synthetic Data Generation for Linear Regression

While adding an extra term penalizes the weight matrix from becoming too large, why does this prevent overfitting? This is because if lambda becomes very large the weight of the matrices is incentivized to be close to 0. In other words, you are zeroing the impact of hidden units so that a simpler network is the result. The hidden units are still there, but now they have a much smaller effect. With a small W and and small Z, the activation function will likely be fairly linear. 


Why does regularization prevent overfitting? 

You define 0.1% as “human-level performance.” After working further on your algorithm, you end up with the following: 

- Human-level performance: 0.1%
- Training set error: 2.0%
- Dev set error: 2.1%

Human Level Performance: Based on the evidence below, which two of the following four options seem the most promising to try? 


$\bullet$ Local constancy and smoothness priors are mostly used "priors". they are the methods to ensure the learned functions are locally constant or smooth. They need to be f(x)=f(x+c). For example, k-nearest neighbor copy directly  data from nearby while most kernel interpolate between nearby data.

$\bullet$ The smoothness works best for function with enough examples and do not have lots of dimensions. However, there are methods to represent and estimate complicated functions.

$\bullet$ Complicated functions can be  estimated and generalized well- by dividing regions and use underlying assumption to associate each regions. For example, a function with O(2^k) regions can be defined by O(k) examples and use additional assumptions to connect these regions(Chapter 2.5.11).


Local Constancy and Smoothness Priors

While a parameter norm penalty is one way to regularize parameters to be close to one another, the more popular way is to use constraints: to force sets of parameters to be equal. This method of regularization is often referred to as parameter sharing, because we interpret the various models or model components as sharing a unique set of parameters. A signiﬁcant advantage of parameter sharing over regularizing the parameters to be close (via a norm penalty) is that only a subset of the parameters (the unique set) needs to be stored in memory. In certain models—such as the convolutional neural network—this can lead to signiﬁcant reduction in the memory footprint of the model.



	Parameter sharing: to force sets of parameters to be equal

	Main application: Convolutional Neural Network (CNN)

	Advantages: signiﬁcant reduction in the memory footprint of the model




Parameter Sharing

Parameter tying refers to the fact that the two models, A and B, have a certain correlation. For example, both models are doing a classification task. Based on this prior knowledge, we can assume that the parameters of model A and model B should be similar, and then we can use this information through regularization.

Parameter Tying

What do L1 regularization and L2 regularization have in common?

L1 regularization and L2 regularization

MTL acts as a regularizer by introducing an inductive bias. As such, it reduces the risk of overfitting and improves its ability to fit random noise.

MTL as a Regularizer

Classical regularization techniques mitigate overfitting by adding a parameter norm penalty to the loss function during training. This extra penalty term restricts the capacity of the model, thereby keeping the learned model simple and reducing its overall complexity. Different parameter norm penalties lead to different regularization behaviors and preferred solutions.

Learn Before

Related

Learn After