Laplace distribution also called double exponential distribution has a sharp bend at random values of  $\mu$

$$Laplace(x;\mu,\gamma)= \frac{1}{2 \gamma} e^{\frac{ - \lvert x - \mu \rvert}{\gamma}}$$


San Jose State University

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Common Probability Distributions

Unlike $$\ell_2$$ regularization which distributes weights evenly, $$\ell_1$$ regularization penalizes the absolute values of the weights. This leads to models that concentrate weights on a small set of features by clearing the other weights to zero, making it an effective method for feature selection. If a model only relies on a few features, it may eliminate the need to collect, store, or transmit data for the dropped features. Linear models that are $$\ell_1$$-regularized are popularly known as lasso regression.

L1 Regularization

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

Laplace Distribution

The curve of probability density function of exponential distribution  has a sharp bend at x=0.

Probability density function of exponential distribution is given by

$$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$

For all negative values of x, $\mathbf{1}_{x\geq0}$ an indicator function assigns zero probability.

Exponential Distribution

Inorder to showcase a probability distribution where the mass can be concentrated at a single point,the probability density function is defined by dirac delta function $\delta(x)$.

The dirac delta function is zero everywhere except at x=0 and it has an area of one.

p{x) = $\delta(x-\mu)$,this equation explains that the function p(x) is having infinitemosuly large value at $x=\mu$ with unit area


Dirac Distribution

Empirical distribution  showcases the empirical frequency associated with a point/sample in the dataset/set of samples.

$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta (x-x^i)$

The dirac delta function is used to describe the empirical distribution of continuous random variables.

Multinoulli distribution is sufficient to explain the empirical distribution of discrete random variables.


Empirical Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

Lasso Regression is similar to ridge regression, but instead we use the L1 regularization penalty.
This has the effect of setting some coefficient estimates to exactly zero for the least influential variables, which leads to models only including a subset of variables. This is called a space solution, which is a kind of feature selection.
The parameter $\alpha$ controls the amount of L1 regularization (default = 1.0).
The prediction formula is the same as OLS.

$$RSS_{LASSO}(\beta_{j}, \beta_{0}) =
\sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert=RSS+\lambda\sum_{j=1}^{p}\vert{\beta}_{j}\vert.
$$ 

Lasso Regression

The trade-off between the standard prediction loss and the additive weight decay penalty is characterized by the regularization constant, $$\lambda$$. This nonnegative hyperparameter is fit using validation data and modifies the objective to $$L(\mathbf{w}, b) + \frac{\lambda}{2} \|\mathbf{w}\|^2$$. When $$\lambda = 0$$, the original loss function is recovered. For $$\lambda > 0$$, the size of the weights is restricted, with larger values of $$\lambda$$ constraining the weights more considerably. The penalty term is divided by $$2$$ by convention so that the constant cancels out gracefully when the derivative of the quadratic function is taken.

Learn Before

Related