Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

An empirical distribution is a probability distribution that places an equal probability mass of $$\frac{1}{m}$$ on each of the $$m$$ data points in a given dataset. For continuous random variables, it is defined using the Dirac delta function as $$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta(x-x^{(i)})$$. For discrete random variables, a Multinoulli distribution is sufficient to model the empirical distribution of the observed frequencies.

Empirical Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

To specify a probability distribution where all mass is concentrated at a single point, the probability density function is defined using the Dirac delta function, $$\delta(x)$$. The Dirac delta function evaluates to zero everywhere except at $$x=0$$, and it integrates to one. A Dirac distribution centered at $$\mu$$ is given by the formula $$p(x) = \delta(x-\mu)$$. This indicates that $$p(x)$$ is infinitely large at $$x=\mu$$ and has a unit area.

Dirac Distribution

The curve of the probability density function for the exponential distribution has a sharp bend at $$x=0$$. The probability density function is given by $$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$, where $$\mathbf{1}_{x\geq0}$$ is an indicator function that assigns zero probability for all negative values of $$x$$.

Probability Density Function of the Exponential Distribution

The Laplace distribution, also known as the double exponential distribution, is a probability distribution that allows us to place a sharp peak of probability mass at an arbitrary point $$\mu$$. Its probability density function is given by the formula: $$\text{Laplace}(x; \mu, \gamma) = \frac{1}{2 \gamma} \exp\left( - \frac{\lvert x - \mu \rvert}{\gamma} \right)$$.

Laplace Distribution

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Johns Hopkins University

- Combinatorics
- Probability Distributions
- Random Variables
- Independence
- Chain Rules

Topics in Probability Theory

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Combinatorics is the study of how to count the various combinations, permutations, subgroups, etc. that can be formed from a given set of objects/people/etc. Knowledge of combinatorics is vital for an understanding of probability because when calculating the probability of an event one needs to know the number of potential ways an event can occur. 

Combinatorics

There are 3 possible sources of uncertainty:
- Inherent stochasticity
- Incomplete observability
- Incomplete modeling

Uncertainty 

Common Probability Distributions

This kind of probability deals with the rates at which certain even occur.

Frequentist Probability

When we want to calculate the probability of a subset of variables, we use marginal probability. For discrete variables, marginal probability is calculated by summing over other variables to find the probability of a specific variable. For continuous variables, marginal probability is calculated by performing integration instead of summation. 

Marginal Probability

Conditional probability is the likelihood of an event occurring, given that another event has already occurred. The event that is given must have a non-zero probability. This relationship is commonly denoted as $P(A|B)$ or $Pr(A|B)$, which reads as 'the probability of A given B'. More generally, it can be represented as $Pr(\cdot|x_1, ..., x_n)$ or simply $p(\cdot|x_1, ..., x_n)$, indicating the probability distribution of a variable conditioned on the values of other variables $x_1, ..., x_n$.

Conditional Probability

Any joint probability can be decomposed into conditional probabilities using Chain Rule. For example: 
$$P(a, b, c) = P(a| b, c) \times P(b, c) =  P(a | b, c) \times P(b | c) \times P(c)$$

Chain Rule

This kind of probability deals with the qualitative levels of certainty of various propositions.


Bayesian Probability

A variable that holds different values by chance as a result of a random experiment.

Random Variable

Population distributions are generally used to describe the probability of a random variable or set of random variables taking on each of its possible states.

Probability Distribution

If two random variables $$X$$ and $$Y$$ are independent, the occurrence of one does not affect the other: $$P(X, Y) = P(X) \times P(Y)$$ If $$X$$ and $$Y$$ are conditionally independent given a variable $$Z$$, they become independent once the value of $$Z$$ is known: $$P(X, Y | Z) = P(X | Z) \times P(Y | Z)$$

Learn Before

Related

Learn After