$ \theta_{kl}= p(x_k = l)$
$ \theta_kl = \frac {n_{kl}}{N}$ 
Number of times feature k takes value l in dataset




Maximum Likelihood

Independent Features

The formula $\mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\text{gold}})$ represents a loss function that quantifies the difference between a model's predicted probability distribution, $\mathbf{p}_{i+1}^{\theta}$, and the ground truth or "gold" probability distribution, $\mathbf{p}_{i+1}^{\text{gold}}$. The predicted distribution is parameterized by $\theta$, which are the model's parameters that are updated during training. The goal of training is typically to minimize this loss, thereby making the predicted distribution as close as possible to the true distribution. The subscript $i+1$ suggests this is often used in sequential contexts, like predicting the next element in a sequence.

Loss Function for Predicted vs. Gold Probability Distributions

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

San Diego State University

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

If the model is based on fixed calculations we will have the same results every time(i.e. taking average value of each pixel), thus that's not generative. It is inevitable to use random element influencing each samples that are produced by our model. 
(**quick note** this comparison can be  similar to *linear* vs *non-linear* activation functions in deep learning)

*Probabilistic* rather than *Deterministic*

Most problems that you might have faced in the machine learning are discriminative in their nature. Unlike generative modeling here each sample has a **label** .  You can consider discriminative modeling as supervised learning.

The discriminative model learns the decision function f(X) directly from the data, or the conditional probability distribution P(Y|X) as the prediction model. The discriminative model is concerned with what output Y should be predicted for a given input X. Common discriminative models are: K-nearest neighbor, perceptron, linear regression, linear discriminant analysis (LDA), LR, SVM, decision tree, neural network, boosting, conditional random field, maximum entropy model.

In order to understand difference between *discriminative* and *generative* modeling you can use these definitions:
 

Discriminative Modeling

1. Complete understanding of how the data was generated, not merely categorizing objects.

2. Seems highly promising in the Reinforcement Learning 

3. It is mightier to acquire machine with  the form of human intelligence. 

Why *Generative Modeling* ? 

Quick Recap For Some Probability Concepts

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

1. Variational AutoEncoders (_VAE_)
2. Generative Adversarial Networks (_GAN_)
3. Diffusion Inversion
4. Approximate Bayesian computation (_ABC_) framework

Generative Modeling Architectures

https://www.amazon.com/Generative-Deep-Learning-Teaching-Machines/dp/1492041947

David Foster's Generative Deep Learning

- DBNs were one of the first nonconvolutional models to successfully admit training of deep architectures
- They are generative models with several layers of latent/hidden variables in which the latent variables are typically binary, while the visible units may be binary or real.
- Every unit in each layer is connected to every unit in each neighboring layer.
- The probability distribution represented by the DBN is given by
$$P(h^{(l)}, h^{(l-1)}) \propto exp(b^{{(l)}^\top}h^{(l)} + b^{(l-1)\top}h^{(l-1)} + h^{(l-1)\top}W^{(l)}h^{(l)})$$
$$P(h_{i}^{(k)}=1 | h^{(k+1)}) = \sigma (b_{i}^{(k)}+W_{:,i}^{(k+1)\top}h^{(k+1)})\forall i, \forall k \in 1, ..., l-2$$
$$P(v_i=1 | h^{(1)}) = \sigma (b_i^{(0)}+W_{:,i}^{(1)\top}h^{(1)})\forall i$$
- In the case of real-valued visible units, we substitute
$$\mathbf{v}\sim\mathcal{N}(v;b^{(0)}+W^{(1)\top}h^{(1)},\beta^{-1})$$
- with $\beta$ diagonal for tractability


Deep Belief Networks (DBNs)

Practitioners usually evaluate generative models by having experimental subjects visually assess the samples. This can help to notice under and overfitting, but struggles because bad models may still look solid visually. In this case, databases like MNIST are used to continue to check.

Evaluating Generative Models

Generative Adversarial Networks are a type of generative modeling approach. In this process, the generator network competes with the discriminator network, which attempts to distinguish between samples drawn from either the training data or the generator. This allows the discriminator to produce a probability that a particular training example is real rather than a fake sample drawn from the model. 

Generative Adversarial Networks

A generator network using a convolutional structure is often useful for generating images. We utilize the "transpose" of the convolution operator, often yielding more realistic images using fewer parameters.

Convolutional Generative Networks

A **Generative Stochastic Network**, or **GSN**, is a generative model with latent or hidden variables $\bold h$ in the Markov chain, in addition to the ordinary visible random variables $\bold x$.

Generative Stochastic Networks (GSNs)

Here is the example to help make the idea of a generative model comprehensible for you. Suppose you have dog images and you want to generate a dog image that looks real but that dog actually doesn't exist. This can be achieved by learning the characteristics of dog images. This is a typical problem which we can solve using generative modeling.

Here is a visual clue for you:

Generative Model Example

•	The standard procedure for drawing samples from a normal distribution with mean µ and covariance Σ  is to feed samples z from a normal distribution with zero mean and identity covariance into a very simple generator. 
•	Pseudorandom number generators can also use nonlinear transformations of simple distributions. In this case, we can use the inverse transform sampling techniques. If we can specify p(x), integrate over x, and invert the resulting function. In this case, we don’t need to use machine learning methods.


How to generate samples from not complicated distributions using generator networks?

• To generate samples from more complicated distributions that are diﬃcult to specify directly, diﬃcult to integrate over, or whose resulting integrals are diﬃcult to invert, we use a feedforward network to represent a parametric family of nonlinear functions g, and use training data to infer the parameters selecting the desired function

Generate samples from complicated distributions

• When emitting the parameters of a conditional distribution over x, it can generate discrete data as well as continuous data. However, these data could be algebraically manipulated by humans, which is a weakness.

• On the other hand, when the generator net provides samples directly, it can generate only continuous data, and are not forced to use conditional distribution. The problem with direct sampling is that the model could no longer be trained using back-propagation. 


Emitting the parameters of a conditional distribution versus directly emitting samples

•The learning process of generative modeling requires optimizing intractable criteria. In the context of diﬀerentiable generator nets, the criteria are intractable because the data does not specify both the inputs z and the outputs x of the generator net. 	

•IHowever, in the case of supervised learning, both the inputs x and the outputs y were given, and the optimization procedure needs only to learn how to produce the speciﬁed mapping.

Why is Generative modeling more difficult than classiﬁcation or regression

Some generative models allow the probability density function to be evaluated explicitly; others do not. Some are structured probabilistic models which can be described in terms of graphs and factors; others cannot, though they still represent probability distributions.

Variations of generative models

Which of the following models are machine learning generative models?

Learn Before

Related

Learn After