Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

Reference for Variational AutoEncoder Code

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Limitation(s) of Variational Autoencoder Framework


In a standard **autoencoder**, each input (such as an image) is mapped to a single point in the latent space. In contrast, in a **variational autoencoder** (VAE), each input is mapped to a multivariate normal distribution around a mean point in that space. The following diagram illustrates this difference:

Autoencoder Encoder vs. Variational Autoencoder Encoder

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

San Diego State University

1. Variational AutoEncoders (_VAE_)
2. Generative Adversarial Networks (_GAN_)
3. Diffusion Inversion
4. Approximate Bayesian computation (_ABC_) framework

Generative Modeling Architectures

Variational AutoEncoders

This generative model scheme is set apart from MCMC sampling and ancestral sampling because it is based on nonequilibrium thermodynamics. This approach assumes that the probabilistic distribution we will sample from has some structure. This structure is gradually broken down by some diffusion that increases the entropy in the distribution thus making it unstructured. 


To create the generative model, we then train a model that reverses the process and gradually returns structure to our distribution. Similarly to MCMC, we apply many iterations to produce our sample; however, in this case our model is iterating in order to transform the entropy-filled distribution to our target distribution.   


Diffusion Inversion

The ABC approach rejects or modifies samples to make the moments of the selected samples match the desired distribution. Unlike moment matching, this approach does not train the mode to automatically produce samples with the correct moments but rather it modifies the sample directly. A simple explanation is given below: 

Learn Before

Related

Learn After