An empirical distribution is a probability distribution that places an equal probability mass of $$\frac{1}{m}$$ on each of the $$m$$ data points in a given dataset. For continuous random variables, it is defined using the Dirac delta function as $$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta(x-x^{(i)})$$. For discrete random variables, a Multinoulli distribution is sufficient to model the empirical distribution of the observed frequencies.

San Jose State University

Google

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Common Probability Distributions

To specify a probability distribution where all mass is concentrated at a single point, the probability density function is defined using the Dirac delta function, $$\delta(x)$$. The Dirac delta function evaluates to zero everywhere except at $$x=0$$, and it integrates to one. A Dirac distribution centered at $$\mu$$ is given by the formula $$p(x) = \delta(x-\mu)$$. This indicates that $$p(x)$$ is infinitely large at $$x=\mu$$ and has a unit area.

Dirac Distribution

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

Empirical Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

The curve of the probability density function for the exponential distribution has a sharp bend at $$x=0$$. The probability density function is given by $$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$, where $$\mathbf{1}_{x\geq0}$$ is an indicator function that assigns zero probability for all negative values of $$x$$.

Probability Density Function of the Exponential Distribution

The Laplace distribution, also known as the double exponential distribution, is a probability distribution that allows us to place a sharp peak of probability mass at an arbitrary point $$\mu$$. Its probability density function is given by the formula: $$\text{Laplace}(x; \mu, \gamma) = \frac{1}{2 \gamma} \exp\left( - \frac{\lvert x - \mu \rvert}{\gamma} \right)$$.

Learn Before

Related