If we have $\mu$ mean and symmetric covariance matrix C in 2D they are defined as:
$\mu$ = $$\begin {bmatrix}
   \mu_1 \\
   \mu_2 \\
\end {bmatrix}$$

$C$ = $$\begin {bmatrix}
\sigma_{1}^{2} & r \sigma_1 \sigma_2 \\
r \sigma_1 \sigma_2 & \sigma_{2}^{2}\\
\end {bmatrix}$$ 

In Variational AutoEncoder is assumed that there is no correlation between these dimensions.  This means that encoder needs to map each input to mean and variance vectors. So, as the result image will be encoded into two vectors:
1. **mu** -  the mean point distribution.
2. **log_var** - the logarithm of the variance of each dimensions.
In order to encode image into a specific point we can use this formula:
$$ z = mu + exp(log\_var / 2) $$

Variational AutoEncoder Encoder

In **autoencoder ** each image is simply mapped to one point in the latent space, while in **variational autoencoder** each image is mapped to multivariate normal distribution around that point.
Here is the image that demonstrates this difference:


San Diego State University

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

AutoEncoder Encoder vs Variational AutoEncoder Encoder

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

Reference for Variational AutoEncoder Code

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Learn Before

Related

Learn After