Introduction to Generative Adversarial Networks (GANs): Types, and Applications, and Implementation
https://heartbeat.fritz.ai/introduction-to-generative-adversarial-networks-gans-35ef44f21193

Introduction to Generative Adversarial Networks (GANs): Types, and Applications, and Implementation

DCGANs are an improvement of GANs. They are more stable and generate higher quality images. In DCGAN, batch normalization is done in both networks, i.e the generator network and the discriminator network. They can be used for style transfer. For example, you can use a dataset of handbags to generate shoes in the same style as the handbags.

Deep Convolutional GANs (DCGANs)

These GANs use extra label information and result in better quality images and are able to control how generated images will look. cGANs learn to produce better images by exploiting the information fed to the model.

Conditional GANs (cGANs)

The authors of this paper propose a solution to the problem of synthesizing high-quality images from text descriptions in computer vision. They propose Stacked Generative Adversarial Networks (StackGAN) to generate 256x256 photo-realistic images conditioned on text descriptions. They decompose the hard problem into more manageable sub-problems through a sketch-refinement process.

StackGAN

InfoGAN is an information-theoretic extension to the GAN that is able to learn disentangled representations in an unsupervised manner. InfoGANs are used when your dataset is very complex, when you’d like to train a cGAN and the dataset is not labelled, and when you’d like to see the most important features of your images.

InfoGANs

WGANs change the loss function to include a Wasserstein distance. They have loss functions that correlate to image quality.

Wasserstein GANs(WGAN)

The authors of this paper propose a method based on generative adversarial networks that learns to discover relations between different domains. Using the discovered relations, the network transfers style from one domain to another. In the process, it preserves key attributes such as orientation and face identity.


Discover Cross-Domain Relations with Generative Adversarial Networks(Disco GANS)

1. Deep Convolutional GANs (DCGANs)
2. Conditional GANs (cGANs)
3. StackGAN
4. InfoGANs
5. Wasserstein GANs(WGAN)

University of Michigan - Ann Arbor

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Networks (PDF). Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014). pp. 2672–2680.

Generative Adversarial Networks

A known dataset serves as the initial training data for the discriminator, presenting it with samples from the training dataset, until it achieves acceptable accuracy. Typically the generator is seeded with randomized input that is sampled from a predefined latent space (e.g. a multivariate normal distribution). Thereafter, candidates synthesized by the generator are evaluated by the discriminator. Independent backpropagation procedures are applied to both networks so that the generator produces better images, while the discriminator becomes more skilled at flagging synthetic images. The generator is typically a deconvolutional neural network, and the discriminator is a convolutional neural network.

Method of GAN

GAN applications have increased rapidly:

1. Fashion, art and advertising.

2. Science.

3. Video games.

4. Miscellaneous applications.

Applications of GAN

Different types of GANs

Auto-Regressive Networks are a kind of directed probability graph. The conditional probability is described using neural network and chain rule. The key defination of the  Auto-Regressive Network is:
$p(x) =\prod_{i=1}^{n_i} p(x_i|x_{i-1},...,x_1)$

Auto-Regressive Network

Restricted Boltzmann Machines (RBM), Deep Boltzmann Machines (DBM), and variants:
- Undirected graphical models with latent variables instead of directed

Deep Belief Networks (DBN):
- Single undirected layer and multiple directed layers

Score Matching and Noise-contrastive Estimation (NCE):
- Don't approximate or bound log-likelihood

Generative Stochastic Network (GSN):
- Don’t involve defining a probability distribution explicitly; instead trains a generative machine to draw samples from the desired distribution


Related Work to GANs

Advantages:
- No need for Markov chains
- Handle sharp distributions well compared to Markov chain methods
- Only back propagation is used to obtain gradients
- No inference needed during learning

Disadvantages:
- No explicit $p_g(x)$
- Necessary to avoid over-training of generative model without updating discriminative model

Advantages/Disadvantages of Gans

When tested against three different generative models using the same starting dataset and evaluated using log-likelihood estimates, adversarial nets produce acceptable results to stay competitive with existing models. 

Below are some of the challenges/benefits that some generative models encounter while testing:

GANs tested performance

Legend:
- $\bold{z}$: Noise distribution (mapped to $\bold{x}$).
- $\bold x$: Data distribution.
- **Blue**: Discriminative Distribution.
- **Green**: Generative Distribution.
- **Black**: Data Distribution.

The discriminative distribution is repeatedly updated to minimize the difference between our generative distribution and true data distribution (shown in figure left to right) so the generative distribution becomes a more accurate approximation of what the data shows.

Illustration of GAN's

Updating Algorithm for GAN's

- Proposition 1: Optimal Discriminator $D$ for given generator $G$ is $D_g^*=\frac {p_{data}(\bold x)} {p_{data}(\bold x) + p_g(\bold x)}$.
- Theorem 1: The global minimum of virtual training criterion $C(G)$ is found if and only if $p_g=p_{data}$, having value $-\log 4$.
- Proposition 2: $p_g$ converges to $p_{data}$ if: 
1. 	$G$ and $D$ have enough capacity,
2. At each step of algorithm 1 the discriminator is allowed to reach its optimum given $G$,
3. $p_g$ is updated to improve the criterion $\mathbb{E}_{\bold x ~ p_{data}}[\log D_G^*(\bold x)] + \mathbb{E}_{\bold x ~ p_g}[\log(1-D_G^*(\bold x)]$.


Learn Before

Related

Learn After