An empirical distribution is a probability distribution that places an equal probability mass of $$\frac{1}{m}$$ on each of the $$m$$ data points in a given dataset. For continuous random variables, it is defined using the Dirac delta function as $$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta(x-x^{(i)})$$. For discrete random variables, a Multinoulli distribution is sufficient to model the empirical distribution of the observed frequencies.

Empirical Distribution

To specify a probability distribution where all mass is concentrated at a single point, the probability density function is defined using the Dirac delta function, $$\delta(x)$$. The Dirac delta function evaluates to zero everywhere except at $$x=0$$, and it integrates to one. A Dirac distribution centered at $$\mu$$ is given by the formula $$p(x) = \delta(x-\mu)$$. This indicates that $$p(x)$$ is infinitely large at $$x=\mu$$ and has a unit area.

San Jose State University

Google

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Common Probability Distributions

A probability density function characterizes continuous random variables by assigning a probability density, rather than an exact probability, to possible values. Since the exact probability of any single continuous value is zero, determining the probability that the variable falls within a specific interval requires calculating the definite integral of the density function over that interval.

Probability Density Function

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

Dirac Distribution

The curve of the probability density function for the exponential distribution has a sharp bend at $$x=0$$. The probability density function is given by $$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$, where $$\mathbf{1}_{x\geq0}$$ is an indicator function that assigns zero probability for all negative values of $$x$$.

Probability Density Function of the Exponential Distribution

The Laplace distribution, also known as the double exponential distribution, is a probability distribution that allows us to place a sharp peak of probability mass at an arbitrary point $$\mu$$. Its probability density function is given by the formula: $$\text{Laplace}(x; \mu, \gamma) = \frac{1}{2 \gamma} \exp\left( - \frac{\lvert x - \mu \rvert}{\gamma} \right)$$.

Learn Before

Related

Learn After