If the model is based on fixed calculations we will have the same results every time(i.e. taking average value of each pixel), thus that's not generative. It is inevitable to use random element influencing each samples that are produced by our model. 
(**quick note** this comparison can be  similar to *linear* vs *non-linear* activation functions in deep learning)

*Probabilistic* rather than *Deterministic*

Most problems that you might have faced in the machine learning are discriminative in their nature. Unlike generative modeling here each sample has a **label** .  You can consider discriminative modeling as supervised learning.

The discriminative model learns the decision function f(X) directly from the data, or the conditional probability distribution P(Y|X) as the prediction model. The discriminative model is concerned with what output Y should be predicted for a given input X. Common discriminative models are: K-nearest neighbor, perceptron, linear regression, linear discriminant analysis (LDA), LR, SVM, decision tree, neural network, boosting, conditional random field, maximum entropy model.

In order to understand difference between *discriminative* and *generative* modeling you can use these definitions:
 

Discriminative Modeling

1. Complete understanding of how the data was generated, not merely categorizing objects.

2. Seems highly promising in the Reinforcement Learning 

3. It is mightier to acquire machine with  the form of human intelligence. 

Why *Generative Modeling* ? 

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

Quick Recap For Some Probability Concepts

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

1. Variational AutoEncoders (_VAE_)
2. Generative Adversarial Networks (_GAN_)
3. Diffusion Inversion
4. Approximate Bayesian computation (_ABC_) framework

Generative Modeling Architectures

https://www.amazon.com/Generative-Deep-Learning-Teaching-Machines/dp/1492041947

David Foster's Generative Deep Learning

- DBNs were one of the first nonconvolutional models to successfully admit training of deep architectures
- They are generative models with several layers of latent/hidden variables in which the latent variables are typically binary, while the visible units may be binary or real.
- Every unit in each layer is connected to every unit in each neighboring layer.
- The probability distribution represented by the DBN is given by
$$P(h^{(l)}, h^{(l-1)}) \propto exp(b^{{(l)}^\top}h^{(l)} + b^{(l-1)\top}h^{(l-1)} + h^{(l-1)\top}W^{(l)}h^{(l)})$$
$$P(h_{i}^{(k)}=1 | h^{(k+1)}) = \sigma (b_{i}^{(k)}+W_{:,i}^{(k+1)\top}h^{(k+1)})\forall i, \forall k \in 1, ..., l-2$$
$$P(v_i=1 | h^{(1)}) = \sigma (b_i^{(0)}+W_{:,i}^{(1)\top}h^{(1)})\forall i$$
- In the case of real-valued visible units, we substitute
$$\mathbf{v}\sim\mathcal{N}(v;b^{(0)}+W^{(1)\top}h^{(1)},\beta^{-1})$$
- with $\beta$ diagonal for tractability


Deep Belief Networks (DBNs)

Practitioners usually evaluate generative models by having experimental subjects visually assess the samples. This can help to notice under and overfitting, but struggles because bad models may still look solid visually. In this case, databases like MNIST are used to continue to check.

Evaluating Generative Models

Generative Adversarial Networks are a type of generative modeling approach. In this process, the generator network competes with the discriminator network, which attempts to distinguish between samples drawn from either the training data or the generator. This allows the discriminator to produce a probability that a particular training example is real rather than a fake sample drawn from the model. 

Generative Adversarial Networks

A generator network using a convolutional structure is often useful for generating images. We utilize the "transpose" of the convolution operator, often yielding more realistic images using fewer parameters.

Convolutional Generative Networks

A **Generative Stochastic Network**, or **GSN**, is a generative model with latent or hidden variables $\bold h$ in the Markov chain, in addition to the ordinary visible random variables $\bold x$.

Generative Stochastic Networks (GSNs)

Here is the example to help make the idea of a generative model comprehensible for you. Suppose you have dog images and you want to generate a dog image that looks real but that dog actually doesn't exist. This can be achieved by learning the characteristics of dog images. This is a typical problem which we can solve using generative modeling.

Here is a visual clue for you:

Generative Model Example

•	The standard procedure for drawing samples from a normal distribution with mean µ and covariance Σ  is to feed samples z from a normal distribution with zero mean and identity covariance into a very simple generator. 
•	Pseudorandom number generators can also use nonlinear transformations of simple distributions. In this case, we can use the inverse transform sampling techniques. If we can specify p(x), integrate over x, and invert the resulting function. In this case, we don’t need to use machine learning methods.


How to generate samples from not complicated distributions using generator networks?

• To generate samples from more complicated distributions that are diﬃcult to specify directly, diﬃcult to integrate over, or whose resulting integrals are diﬃcult to invert, we use a feedforward network to represent a parametric family of nonlinear functions g, and use training data to infer the parameters selecting the desired function

Generate samples from complicated distributions

• When emitting the parameters of a conditional distribution over x, it can generate discrete data as well as continuous data. However, these data could be algebraically manipulated by humans, which is a weakness.

• On the other hand, when the generator net provides samples directly, it can generate only continuous data, and are not forced to use conditional distribution. The problem with direct sampling is that the model could no longer be trained using back-propagation. 


Emitting the parameters of a conditional distribution versus directly emitting samples

•The learning process of generative modeling requires optimizing intractable criteria. In the context of diﬀerentiable generator nets, the criteria are intractable because the data does not specify both the inputs z and the outputs x of the generator net. 	

•IHowever, in the case of supervised learning, both the inputs x and the outputs y were given, and the optimization procedure needs only to learn how to produce the speciﬁed mapping.

Why is Generative modeling more difficult than classiﬁcation or regression

Some generative models allow the probability density function to be evaluated explicitly; others do not. Some are structured probabilistic models which can be described in terms of graphs and factors; others cannot, though they still represent probability distributions.

Variations of generative models

Which of the following models are machine learning generative models?

Generative models

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

University of Massachusetts at Amherst

University of Michigan - Ann Arbor

San Diego State University

Deep learning algorithms can be implemented to solve real-world problems today.

Deep feedforward networks, also called multilayer perceptrons (MLPs), is the quintessential deep learning model that is behind nearly all modern practical applications of deep learning.

Scaling these models to large inputs such as high-resolution images or long temporal sequences requires specialization. 
 - Convolutional neural networks (CNN) can be used for processing large images.
 - Recurrent neural networks (RNN) can be used for processing temporal sequences. 


Deep Learning Algorithms

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

Generative Models

There are functions you can compute with a small layer that shallower networks would need a lot more hidden units in order to compute. 

Circuit Theory

https://www.simplilearn.com/deep-learning-algorithms-article

More about deep learning algorithms 

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

What can convolutional neural networks be used for?

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Whereas deep learning algorithms were once always implemented on a single computer's central processing unit (CPU), research has shown that families of CPUs or even a single graphics processing unit (GPU) is much more effective. Some systems employ even greater parallelization to achieve the fastest training times with distributed architectures.

Implementations of Deep Learning

Monte Carlo algorithms return results with variability. The variability can typically be reduced by allocating more resources. The benefit of Monte Carlo algorithms is that they can provide an approximated estimation with limited source while maintaining a certain degree of accuracy. In Machine Learning, many problems are too complicated to have precise answers. Monte Carlo approximations are therefore a good approach to these problems.

Monte Carlo Methods

Deep learning frameworks provide specialized software tools that streamline the construction, training, and deployment of neural networks. The availability of these efficient frameworks has made the design and implementation of whole system optimization significantly easier, which is a critical component in achieving high performance for complex deep learning pipelines.

Deep Learning Frameworks

An adversarial example is an input to a machine learning model that has been intentionally modified with small, often imperceptible perturbations designed to cause the model to output an incorrect classification. For instance, adding specific, calculated noise to an image of a panda can cause a deep neural network to misclassify it as a gibbon, even though the image remains visually identical to the original for a human observer.

Learn Before

Related

Learn After