The Kullback-Leibler (KL) divergence acts as a regularization term in the loss function of a Variational Autoencoder. It improves the model in two main ways: First, it ensures that the learned latent space follows a well-defined probability distribution. Second, it reduces the probability of gap formation between point clusters in the latent space, encouraging continuous and meaningful representations. The total loss is calculated by adding the reconstruction loss and the KL divergence loss. A weight factor is often applied to the reconstruction loss to balance the two components, and poorly choosing this weight term can lead to unsatisfactory results.

San Diego State University

Google

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

 ```python     
 ### COMPILATION
        def vae_r_loss(y_true, y_pred):
            r_loss = K.mean(K.square(y_true - y_pred), axis = [1,2,3])
            return r_loss_factor * r_loss

        def vae_kl_loss(y_true, y_pred):
            kl_loss =  -0.5 * K.sum(1 + self.log_var - K.square(self.mu) - K.exp(self.log_var), axis = 1)
            return kl_loss

        def vae_loss(y_true, y_pred):
            r_loss = vae_r_loss(y_true, y_pred)
            kl_loss = vae_kl_loss(y_true, y_pred)
            return  r_loss + kl_loss

        optimizer = Adam(lr=learning_rate)
        self.model.compile(optimizer=optimizer, loss = vae_loss,  metrics = [vae_r_loss, vae_kl_loss])```

Learn Before

Related