A “Bernoulli trial” is an experiment or case where the outcome follows a Bernoulli distribution. Some common examples of Bernoulli trials include:

1) A single flip of a coin that may have a head(0) or a tail(1) outcome.

2) A single birth of either a boy(0) or a girl(1).

Bernoulli Trial

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Bernoulli distribution with parameter $\theta$, such that: 
	$$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{i})}$$,
a unbiased estimator for $\theta$ is the mean of those samples:
	$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Bernoulli Parameter

For a Bernoulli distribution characterizing a random variable $$ X $$ with probability $$ p $$ for $$ X=1 $$ ("success") and $$ 1-p $$ for $$ X=0 $$ ("failure"), the key characteristics are:
1) Probability mass function: $$ P(X) = p^{X}(1-p)^{1-X} $$
2) Expectation: $$ E[X] = p(1) + (1-p)(0) = p $$
3) Variance: $$ Var(X) = E[X^2] - (E[X])^2 = p - p^2 = p(1-p) $$

Characteristics of Bernoulli Distribution

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Johns Hopkins University

Claude

Common probability distributions include Normal Distribution, Multinoulli Distribution,  Bernoulli Probability Distribution, Exponential Distribution, and Laplace Distribution. 

Common Probability Distributions

Multi-label classification is a machine learning problem focused on predicting classes that are not mutually exclusive, allowing a single example to be assigned to multiple categories simultaneously. This formulation is particularly useful for auto-tagging tasks, where an item (such as a blog post or an image) might be associated with several correlated tags at once.

Multi-Label Classification

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

Bernoulli Distribution

Laplace distribution also called double exponential distribution has a sharp bend at random values of  $\mu$

$$Laplace(x;\mu,\gamma)= \frac{1}{2 \gamma} e^{\frac{ - \lvert x - \mu \rvert}{\gamma}}$$


Laplace Distribution

Empirical distribution  showcases the empirical frequency associated with a point/sample in the dataset/set of samples.

$\hat{p}(x) = \frac{1}{m} \sum_{i=1}^{m} \delta (x-x^i)$

The dirac delta function is used to describe the empirical distribution of continuous random variables.

Multinoulli distribution is sufficient to explain the empirical distribution of discrete random variables.


Empirical Distribution

Mixture distribution is a combination of several other probability distributions.

Mixture Distribution


There are some functions used while describing probability distribution in deep learning models they are
- Logistic Sigmoid
- Softplus Function


Properties of Common Functions

A prior probability distribution is a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.

prior probability distribution

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

To specify a probability distribution where all mass is concentrated at a single point, the probability density function is defined using the Dirac delta function, $$\delta(x)$$. The Dirac delta function evaluates to zero everywhere except at $$x=0$$, and it integrates to one. A Dirac distribution centered at $$\mu$$ is given by the formula $$p(x) = \delta(x-\mu)$$. This indicates that $$p(x)$$ is infinitely large at $$x=\mu$$ and has a unit area.

Dirac Distribution

The curve of the probability density function for the exponential distribution has a sharp bend at $$x=0$$. The probability density function is given by $$p(x;\lambda)=\lambda \mathbf{1}_{x\geq0} e^{-\lambda x}$$, where $$\mathbf{1}_{x\geq0}$$ is an indicator function that assigns zero probability for all negative values of $$x$$.

Probability Density Function of the Exponential Distribution

A simplified example of Multi-Label Classification is photo classification. This is where a given photo may have multiple objects in the scene and a model may predict the presence of multiple known objects in the photo, such as “bicycle,” “apple,” “person,” etc.

Example of Multi-Label Classification

•	Binary Relevance (first-order)
•	Classifier Chains (high-order)
•	Calibrated Label Ranking (second-order)

Problem Transformation Methods of Multi-Label Classification

Algorithm adaptation is a way to directly apply the existing single label algorithm to multiple labels, mainly including:

Multi-Label KNN：

For each instance, firstly, obtain the K instances closest to it, and then use the label set of these instances to judge the value of the predicted label set of this instance through the maximum a posteriori probability (map).

Multi-Label Decision Tree：

The core of using the decision tree to deal with multi-label content is to give more fine-grained information entropy gain criteria to build this decision tree model.

Learn Before

Related

Learn After