 ```python     
 ### COMPILATION
        def vae_r_loss(y_true, y_pred):
            r_loss = K.mean(K.square(y_true - y_pred), axis = [1,2,3])
            return r_loss_factor * r_loss

        def vae_kl_loss(y_true, y_pred):
            kl_loss =  -0.5 * K.sum(1 + self.log_var - K.square(self.mu) - K.exp(self.log_var), axis = 1)
            return kl_loss

        def vae_loss(y_true, y_pred):
            r_loss = vae_r_loss(y_true, y_pred)
            kl_loss = vae_kl_loss(y_true, y_pred)
            return  r_loss + kl_loss

        optimizer = Adam(lr=learning_rate)
        self.model.compile(optimizer=optimizer, loss = vae_loss,  metrics = [vae_r_loss, vae_kl_loss])```

Variational AutoEncoder Loss Function Code

The Kullback-Leibler (KL) divergence acts as a regularization term in the loss function of a Variational Autoencoder. It improves the model in two main ways: First, it ensures that the learned latent space follows a well-defined probability distribution. Second, it reduces the probability of gap formation between point clusters in the latent space, encouraging continuous and meaningful representations. The total loss is calculated by adding the reconstruction loss and the KL divergence loss. A weight factor is often applied to the reconstruction loss to balance the two components, and poorly choosing this weight term can lead to unsatisfactory results.

Role of Kullback-Leibler (KL) Divergence in the Variational Autoencoder Loss Function

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

San Diego State University

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

In **autoencoder ** each image is simply mapped to one point in the latent space, while in **variational autoencoder** each image is mapped to multivariate normal distribution around that point.
Here is the image that demonstrates this difference:


AutoEncoder Encoder vs Variational AutoEncoder Encoder

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

Variational AutoEncoder Loss Function

https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

Reference for Variational AutoEncoder Code

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Learn Before

Related

Learn After