1. A Generalization of Ridge, Lasso and Elastic Net Regression to Interval Data(https://www.researchgate.net/publication/265905932_A_Generalization_of_Ridge_Lasso_and_Elastic_Net_Regression_to_Interval_Data)
2. Regularization and variable selection via the elastic net(https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.445.4562&rep=rep1&type=pdf)

ElasticNet Regression First Paper

The elastic net method overcomes the limitations of the LASSO (least absolute shrinkage and selection operator) method which uses a penalty function based on L2 . Use of this penalty function has several limitations. For example, in the "large p, small n" case (high-dimensional data with few examples), the LASSO selects at most n variables before it saturates. Also if there is a group of highly correlated variables, then the LASSO tends to select one variable from a group and ignore the others. To overcome these limitations, the elastic net adds a quadratic part to the penalty , which when used alone is ridge regression (known also as Tikhonov regularization). 
The quadratic penalty term makes the loss function strongly convex, and it therefore has a unique minimum. The elastic net method includes the LASSO and ridge regression: in other words, each of them is a special case. Meanwhile, the naive version of elastic net method finds an estimator in a two-stage procedure : first for each fixed {\displaystyle \lambda _{2}}\lambda _{2} it finds the ridge regression coefficients, and then does a LASSO type shrinkage. This kind of estimation incurs a double amount of shrinkage, which leads to increased bias and poor predictions. To improve the prediction performance, the authors rescale the coefficients of the naive version of elastic net by multiplying the estimated coefficients by {\displaystyle (1+\lambda _{2})}(1 + \lambda_2).

analysis and advantage of elasticnet method

1. Support vector machine
2. Metric learning
3. Portfolio optimization
4. Cancer prognosis

Application Areas

In statistics, particularly in the fitting of linear or logistic regression models, the elastic net is a regularized regression method that linearly combines the $$L_1$$ and $$L_2$$ penalties of the lasso and ridge regression methods.

University of Michigan - Ann Arbor

Google

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

Popular Regularization Techniques in Deep Learning

Lasso regression is similar to ridge regression, but instead uses the $$\ell_1$$ regularization penalty. This has the effect of setting some coefficient estimates to exactly zero for the least influential variables, which leads to models only including a subset of variables. This results in a sparse solution, which is a kind of feature selection. The parameter $$\lambda$$ controls the amount of $$\ell_1$$ regularization. The prediction formula is the same as ordinary least squares (OLS), but it minimizes the following objective function: $$RSS_{LASSO}(\beta_{j}, \beta_{0}) = \sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert=RSS+\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert$$

Lasso Regression

Getting more data may be costly and an ineffective way to prevent overfitting. However, you can augment your data to create more. We are adding synthetic data modified from our original data set. You essentially transform your data in a way where it is different form the original, but the data still fits into your given class. It is much more efficient and cost effective. 

Data Augmentation in Deep Learning

Early stopping is a classic regularization technique for deep neural networks that mitigates overfitting by constraining the number of training epochs instead of directly penalizing weight values. This approach is motivated by the fact that neural networks tend to fit clean data before memorizing noisy labels; by halting training at the optimal epoch, the model avoids interpolating noise and thereby improves generalization.

Early Stopping in Deep Learning

Unlike L2 or L1 regularization, Dropout regularization works by assigning a probability P, which is the probability that a given node is turned off for the current iteration of training. By doing this, you make the model simpler while training, which will reduce overfitting, while still gaining the benefits a larger model has during testing. Also, no single node will have excess influence on the model, and the model will spread the weights out, giving us a similar effect to L2 regularization. Generally, Dropout Regularization is the preferred form of regularization, as it not only accomplishes results similar to L2 regularization, it also provides some robustness to the model as each iteration is randomized.

Dropout Regularization in Deep Learning

Which of these techniques are useful for reducing variance (reducing overfitting)?

If your Neural Network model seems to have high variance, what of the following would be promising things to try?

https://medium.com/analytics-vidhya/regularization-in-machine-learning-and-deep-learning-f5fa06a3e58a

Regularization in ML and DL

- Bagging is short for bootstrap aggregating.

- Bagging is a technique for reducing generalization error by combining several machine learning models.

- Bagging employs model averaging.

- This is powerful method of regularization, which is widely used in machine learning contest. However, it is quite impractical, since the computational cost of training several models in expensive.
 


Bagging in Deep Learning

- Instead of training multiple models like bagging, dropout train the ensemble consisting of all subnetworks that can be formed by removing non-output units from an underlying base network. These models share parameters, with each model inheriting a different subset of parameters from parent networks. 

- We need to specify the probability of unit to be included. Typically, the input unit is included with probability 0.8 while that in hidden units is 0.5.

-  Prediction of ensemble is given by geometric mean.


- Cheap computational cost and can be combined with other method or regularization.



Dropout in Deep Learning 

Normalization is one of the techniques used to reduce the bias and variance in statistical learning. It helps to have all the variables centered around 0 and have the same variance.
We subtract all the values of each variable from their mean and divide them by their variance.
$$\mu = \frac{1}{m} \sum_{i = 1}^{m} X^{(i)}$$
$$\sigma^{2} = \frac{1}{m} \sum_{i = 1}^{m} (X^{(i)} - \mu)^2$$
$$X_{New} = \frac{X - \mu}{\sigma}$$

Normalization of Data

- Early attempt to take advantage of the manifold hypothesis
- Nonparametric nearest neighbor algorithm, metric used is derived from the manifolds near which probability concentrates
- Assumes that data on the same manifold all has the same category
- Classifier should be invariant to local factors of variation that correspond to movement. So we use the nearest neighbor distance between two points, which is the distance between the manifolds they belong to
- Cheap alternative on a local level is to approximate each manifold by its tangent plane at a point and measure the distance between the two tangents, or between the tangent plane and a point, by solving a low-dimensional linear system.

Tangent Distance Algorithm

The tangent propagation algorithm is a regularization technique for neural networks that introduces an extra penalty to make the classifier's outputs locally invariant to known factors of variation. These factors correspond to movement along the manifold where examples of the same class concentrate. Local invariance is achieved by requiring the gradient $$\nabla_x f(x)$$ to be orthogonal to known manifold tangent vectors $$v^{(i)}$$ at $$x$$.

Equivalently, the directional derivative of $$f$$ at $$x$$ in the directions $$v^{(i)}$$ is kept small by adding a regularization penalty:
$$\Omega(f) = \sum_{i} ((\nabla_x f(x))^\top v^{(i)})^2$$

The tangent vectors are derived a priori from knowledge of transformations that should not alter the output. While related to data augmentation (its non-infinitesimal counterpart) and the tangent distance algorithm, tangent propagation only regularizes the model against infinitesimal perturbations, which can be challenging for networks using rectified linear units.

Tangent Propagation Algorithm

Eliminates the need to know the tangent vectors a priori. Uses autoencoders to estimate manifold tangent vectors to avoid needing user-specified tangent vectors, which go beyond the classical invariants from the geometry of images and include factors that must be learned because they are object-specific. The algorithm first uses an autoencoder to learn the manifold structure by unsupervised learning, then uses the tangents to regularize a neural net classifier, similar to tangent propagation.

Manifold Tangent Classifier

- Boosting helps to build a strong ensemble( ensemble is a collection of multiple machine learning models) compared to a good capacity( strong learning model) individual machine learning model.

- In the context of deep learning all machine learning models are neural networks.

- Individual neural network is also considered as an ensemble which is improved by incrementally adding hidden layers.

- Output from the ensemble is combined to predict the output.

Boosting in Deep Learning

The no free lunch theorem is proof that no single regularization is strategy is certifiably the best at every task, however that is not to say that every strategy is created equal. There may be no best general strategy, but there may be strategies that are often times more useful than others. That being the case, here is a list of some regularization assumptions/ strategies that have been shown to be generally useful:

- Smoothness
- Linearity
- Multiple explanatory factors
- Causal factors
- Depth/ hierarchical factors
- Shared factors across tasks
- Manifolds
- Natural clustering
- Temporal and spatial coherence
- Sparsity
- Simplicity of factor dependencies.


Appropriate Regularization/ Representation

Weight decay, commonly known as $$\ell_2$$ regularization, is a widely used technique for regularizing parametric machine learning models. Instead of directly manipulating the number of parameters, weight decay operates by restricting the values that the parameters can take. The technique is motivated by the intuition that the simplest function is $$f = 0$$, and the complexity of a linear function, such as $$f(\mathbf{x}) = \mathbf{w}^	op \mathbf{x}$$, can be measured by the distance of its parameters from zero.

Weight Decay

Unlike $$\ell_2$$ regularization which distributes weights evenly, $$\ell_1$$ regularization penalizes the absolute values of the weights. This leads to models that concentrate weights on a small set of features by clearing the other weights to zero, making it an effective method for feature selection. If a model only relies on a few features, it may eliminate the need to collect, store, or transmit data for the dropped features. Linear models that are $$\ell_1$$-regularized are popularly known as lasso regression.

L1 Regularization

Elastic Net Regression

In general, one might expect the Lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size.

However, the number of predictors that is related to the response if never known a priori for real data sets. A technique such as cross-validation can be used to determine which approach is better on a particular data set.

Learn Before

Related

Learn After