While adding an extra term penalizes the weight matrix from becoming too large, why does this prevent overfitting? This is because if lambda becomes very large the weight of the matrices is incentivized to be close to 0. In other words, you are zeroing the impact of hidden units so that a simpler network is the result. The hidden units are still there, but now they have a much smaller effect. With a small W and and small Z, the activation function will likely be fairly linear. 


Why does regularization prevent overfitting? 

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

Popular Regularization Techniques in Deep Learning

You define 0.1% as “human-level performance.” After working further on your algorithm, you end up with the following: 

- Human-level performance: 0.1%
- Training set error: 2.0%
- Dev set error: 2.1%

Human Level Performance: Based on the evidence below, which two of the following four options seem the most promising to try? 


$\bullet$ Local constancy and smoothness priors are mostly used "priors". they are the methods to ensure the learned functions are locally constant or smooth. They need to be f(x)=f(x+c). For example, k-nearest neighbor copy directly  data from nearby while most kernel interpolate between nearby data.

$\bullet$ The smoothness works best for function with enough examples and do not have lots of dimensions. However, there are methods to represent and estimate complicated functions.

$\bullet$ Complicated functions can be  estimated and generalized well- by dividing regions and use underlying assumption to associate each regions. For example, a function with O(2^k) regions can be defined by O(k) examples and use additional assumptions to connect these regions(Chapter 2.5.11).


Local Constancy and Smoothness Priors

While a parameter norm penalty is one way to regularize parameters to be close to one another, the more popular way is to use constraints: to force sets of parameters to be equal. This method of regularization is often referred to as parameter sharing, because we interpret the various models or model components as sharing a unique set of parameters. A signiﬁcant advantage of parameter sharing over regularizing the parameters to be close (via a norm penalty) is that only a subset of the parameters (the unique set) needs to be stored in memory. In certain models—such as the convolutional neural network—this can lead to signiﬁcant reduction in the memory footprint of the model.



	Parameter sharing: to force sets of parameters to be equal

	Main application: Convolutional Neural Network (CNN)

	Advantages: signiﬁcant reduction in the memory footprint of the model




Parameter Sharing

Parameter tying refers to the fact that the two models, A and B, have a certain correlation. For example, both models are doing a classification task. Based on this prior knowledge, we can assume that the parameters of model A and model B should be similar, and then we can use this information through regularization.

Parameter Tying

What do L1 regularization and L2 regularization have in common?

L1 regularization and L2 regularization

MTL acts as a regularizer by introducing an inductive bias. As such, it reduces the risk of overfitting and improves its ability to fit random noise.

MTL as a Regularizer

Classical regularization techniques mitigate overfitting by adding a parameter norm penalty to the loss function during training. This extra penalty term restricts the capacity of the model, thereby keeping the learned model simple and reducing its overall complexity. Different parameter norm penalties lead to different regularization behaviors and preferred solutions.

Parameter Penalties in Classical Regularization

Regularization is a modification made to a learning algorithm with the goal of reducing generalization error, but not training error.

Linear Regression is useful for modeling outputs whose relation to their inputs is linear. However, linear regression performs poorly when trying to model non-linear relationships. We can use regularization to help learning algorithms select certain models over others, particularly to avoid overfitting. 

University of Michigan - Ann Arbor

Bowdoin College

Claude

- **High Avoidable bias**: if it does not fit training set well on cost function (e.g., comparable to human level performance), you might train a bigger network, train longer, switch to a better optimization algorithm, like the Adam, or change the neural network architecture/hyperparameter search.
- **High Variance**: if it does not fit dev set on cost function, you may use regularization, increase the size of training set, or change the neural network architecture/hyperparameter search.
- If it does not fit test set well on cost function, you may change or increase the size of dev set.
- If it does not perform well in real world (e.g., happy users), you may change or increase the size of test set or change the cost function.

Troubleshooting a deep learning model

A "skeptical" prior that, when tuned properly,  slows the rate of learning from the sample. This reduces overfitting while allowing the model to learn the regular features of the sample.



Regularization Priors

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

One reason that people manipulate hyperparameters is to improve the bias and variance that a given model contains. Bias is the difference between the average prediction of our model and the correct value which we are trying to predict. Models with high bias pay little attention to the training data and oversimplify the model, which will lead to high errors on training and test sets. Variance is the variability of model prediction for a given data point or a value which tells us spread of our data. High Variance models do not generalize based on previous data much, if at all, and thus while it may perform well on training data it will not on test sets. Briefly, the Bias vs Variance scale is a scale between over- and under-fitting of a model. By manipulating various hyperparameters, you can change between over- and under-fitting to get the best predictions

Bias and Variance in Deep Learning

When an experienced deep learning engineer works on a new problem, they can usually use insight from previous problems to train a good model on the first try, without needing to iterate multiple times through different models. True/False?

Regularization

The dashed curve is meant to regularize. Since there is more probability around 0, the estimate of 0 shrinks. The other 2 curves are narrower priors and therefore are more skeptical of values further from 0. The thin curve has a standard deviation of 0.5 and the thick curve has a standard distribution of 0.2. 

Learn Before

Related

Learn After