Probability density function plot for a gaussian distribution has got a symmetric bell shape where $\sigma$ defines the width of the curve and $\mu$ describes the center of the curve. The y axis is the probability density function.

The below curve is for a standard normal distribution having $\mu$ = 0  and $\sigma$ = 1


Plot of Normal Distribution Density Function

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

The mathematical formula for the normal distribution's probability density function can be directly implemented in Python. Using standard libraries like `math` and `numpy` (or their deep learning equivalents in PyTorch, JAX, or TensorFlow), a function can be defined to calculate the probability density $$p$$ for a given value $$x$$, given a mean $$\mu$$ and standard deviation $$\sigma$$. The code implementation is as follows:

```python
def normal(x, mu, sigma):
    p = 1 / math.sqrt(2 * math.pi * sigma**2)
    return p * np.exp(-0.5 * (x - mu)**2 / sigma**2)
```

This runnable implementation aligns with the formal definition: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2\right)$$.

Normal Distribution Probability Density Function Code

The Central Limit Theorem is a foundational mathematical principle in probability theory stating that the average of $$n$$ independent random samples drawn from any distribution (with mean $$\mu$$ and standard deviation $$\sigma$$) will approximately follow a normal distribution centered at the true mean $$\mu$$ with a standard deviation of $$\sigma/\sqrt{n}$$, as the sample size $$n$$ grows large. In machine learning, this theorem explains why the empirical error of a classifier evaluated on a test set converges to its true population error at an asymptotic rate of $$\mathcal{O}(1/\sqrt{n})$$.

Central Limit Theorem

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

San Jose State University

Claude

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Common Probability Distributions

Weight decay, commonly known as $$\ell_2$$ regularization, is a widely used technique for regularizing parametric machine learning models. Instead of directly manipulating the number of parameters, weight decay operates by restricting the values that the parameters can take. The technique is motivated by the intuition that the simplest function is $$f = 0$$, and the complexity of a linear function, such as $$f(\mathbf{x}) = \mathbf{w}^	op \mathbf{x}$$, can be measured by the distance of its parameters from zero.

Weight Decay

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

Laplace distribution also called double exponential distribution has a sharp bend at random values of  $\mu$

$$Laplace(x;\mu,\gamma)= \frac{1}{2 \gamma} e^{\frac{ - \lvert x - \mu \rvert}{\gamma}}$$


Laplace Distribution

The curve of probability density function of exponential distribution  has a sharp bend at x=0.

Probability density function of exponential distribution is given by

$$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$

For all negative values of x, $\mathbf{1}_{x\geq0}$ an indicator function assigns zero probability.

Exponential Distribution

Inorder to showcase a probability distribution where the mass can be concentrated at a single point,the probability density function is defined by dirac delta function $\delta(x)$.

The dirac delta function is zero everywhere except at x=0 and it has an area of one.

p{x) = $\delta(x-\mu)$,this equation explains that the function p(x) is having infinitemosuly large value at $x=\mu$ with unit area


Dirac Distribution

Empirical distribution  showcases the empirical frequency associated with a point/sample in the dataset/set of samples.

$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta (x-x^i)$

The dirac delta function is used to describe the empirical distribution of continuous random variables.

Multinoulli distribution is sufficient to explain the empirical distribution of discrete random variables.


Empirical Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

Gaussian (Normal) Distribution

In machine learning _Frobenius_ and _L2_ norms can be used interchangeably. Nevertheless there is slight difference between these two. _L2_ norm in essence is an Euclidean norm with special case, when p=2. More specifically for n-dimensional vector we will have:
   $$L_2 = \sqrt {\textstyle\sum_{i=1}^n x_{i}^{2}}$$.
The _Frobenius_ measures the same thing but in this case we have matrices. That's why in neural networks _L2_ norm is frequently referred as _Forbenius_ norm.

_Frobenius_ and _L2_

In Ridge regression the coefficients and bias are learned using the same least-square criterion, but it adds a penalty for large variations in coefficients; i.e., coefficients are found by minimizing a tuning parameter - which controls the strength of the penalty term.
Once the parameters are learned, the ridge regression prediction formula is the same as OLS.
Ridge regression uses L2 regularization that minimizes the sum of square of coefficients and the influence of the regularization term is controlled by the $\alpha$ parameter. Higher $\alpha$ means more regularization and simpler models.
Use Ridge regression when the number of predictor variables is greater than the number of observations. Below is the formula found in our textbook. 
$$RSS_{RIDGE}(\beta_{j}, \beta_{0}) =
\sum_{i=1}^{n} ({y}_{i}-{\beta}_{0}-\sum_{j=1}^{p}{\beta}_{j}{x}_{ij})^2 +\lambda\sum_{j=1}^{p}{\beta}_{j}^2=RSS+\lambda\sum_{j=1}^{p}{\beta}_{j}^2.
$$ 

Note: Ridge Regression is sensitive to scales of variable. Therefore, we usually standardize the predictors before applying Ridge Regression.

Ridge Regression

What is weight decay?

The trade-off between the standard prediction loss and the additive weight decay penalty is characterized by the regularization constant, $$\lambda$$. This nonnegative hyperparameter is fit using validation data and modifies the objective to $$L(\mathbf{w}, b) + \frac{\lambda}{2} \|\mathbf{w}\|^2$$. When $$\lambda = 0$$, the original loss function is recovered. For $$\lambda > 0$$, the size of the weights is restricted, with larger values of $$\lambda$$ constraining the weights more considerably. The penalty term is divided by $$2$$ by convention so that the constant cancels out gracefully when the derivative of the quadratic function is taken.

Learn Before

Related

Learn After