If we have a mean $$\mu$$ and a symmetric covariance matrix $$C$$ in 2D, they are defined as:

$$\mu = \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix}$$

$$C = \begin{bmatrix} \sigma_{1}^{2} & r \sigma_1 \sigma_2 \\ r \sigma_1 \sigma_2 & \sigma_{2}^{2} \end{bmatrix}$$

In a Variational AutoEncoder, it is assumed that there is no correlation between these dimensions. This means that the encoder needs to map each input to mean and variance vectors. As a result, the image will be encoded into two vectors:

1. **mu** ($$\mu$$) - the mean point distribution.
2. **log_var** - the logarithm of the variance of each dimension.

In order to encode an image into a specific point, we can use this formula:

$$ z = \mu + \exp(log\_var / 2) $$

Variational AutoEncoder Encoder

In a standard **autoencoder**, each input (such as an image) is mapped to a single point in the latent space. In contrast, in a **variational autoencoder** (VAE), each input is mapped to a multivariate normal distribution around a mean point in that space. The following diagram illustrates this difference:

San Diego State University

Google

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

Reference for Variational AutoEncoder Code

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Limitation(s) of Variational Autoencoder Framework


Autoencoder Encoder vs. Variational Autoencoder Encoder

Autoencoders try to train the network so that the weights for the encoder and decoder make the loss between the original data and its reconstruction as small as it is possible. 

What does the Autoencoder try to do ?

```python         
        #<1>The input to the autoencoder is the same as the input to the encoder.
        model_input = encoder_input
        #<2> The output from the autoencoder is the output from the encoder passed 
        # through the decoder.
        model_output = self.decoder(encoder_output)
        
        #<3> The Keras model that defines the full autoencoder—a model that takes 
        # an image, and passes it through the encoder and back out through the 
        # decoder to generate a reconstruction of the original image.
        self.model = Model(model_input, model_output)```

Putting it all together - AutoEncoder Code

1. Sampling for latent space can be very problematic
2. With autoencoders it is not guaranteed that we would obtain the same  quality when sampling from latent space
3. Some generated outputs are  poorly formed

Problems With Autoencoders

https://github.com/davidADSP/GDL_code/blob/master/models/AE.py

Reference to the AutoEncoder Code

__The decoder is a mirror image of the encoder, except instead of convolutional layers, we use convolutional transpose layers, as shown in the code. __

(Actually we don't have to write decoder as a mirror image of the encoder. But it is inevitable for the decoder to have exactly same size for the output image as we the input size of the encoder)
```python
     #<1> define the input to the decoder (the point in the latent space).
     decoder_input = Input(shape=(self.z_dim,), name='decoder_input')

        x = Dense(np.prod(shape_before_flattening))(decoder_input)
        x = Reshape(shape_before_flattening)(x)

        for i in range(self.n_layers_decoder):
            conv_t_layer = Conv2DTranspose(
                filters = self.decoder_conv_t_filters[i]
                , kernel_size = self.decoder_conv_t_kernel_size[i]
                , strides = self.decoder_conv_t_strides[i]
                , padding = 'same'
                , name = 'decoder_conv_t_' + str(i)
                )

            #<2>Stack convolutional transpose layers on top of each other.
            x = conv_t_layer(x)

            if i < self.n_layers_decoder - 1:
                x = LeakyReLU()(x)
                
                if self.use_batch_norm:
                    x = BatchNormalization()(x)
                
                if self.use_dropout:
                    x = Dropout(rate = 0.25)(x)
            else:
                x = Activation('sigmoid')(x)

        decoder_output = x
         
        #<3>The Keras model that defines the decoder—a model that takes a point in t 
        #the latent space and decodes it into the original image domain.
        self.decoder = Model(decoder_input, decoder_output)```

Autoencoder Decoder Code

- Traditionally, autoencoders minimize some function $L(x,g(f(x)))$ where $L$ is a loss function penalizing $g(f(x))$ for being dissimilar from $x$, such as the $L^2$ norm of their difference. 
- A denoising auto encoder (DAE) instead minimizes $L(x,g(f(\widetilde{x})))$ where $\widetilde{x}$ is a copy of $x$ that has been corrupted by some form of noise. Denoising autoencoders must therefore undo this corruption rather than simply copying theirinput.
- Two assumptions are inherent to this approach: 1)Higher level representations are relatively stable and robust to the corruption of the input; 2) To perform denoising well, the model needs to extract features that capture useful structure in the input distribution.
- In other words, denoising is advocated as a training criterion for learning to extract useful features that will constitute better higher level representations of the input.

Denoising Autoencoders

A sparse autoencoder is an autoencoder whose training criterion involves a sparsity penalty $$\Omega(h)$$ on the code layer $$h$$. The loss function is $$L(x, g(f(x))) + \Omega(h)$$, where $$g(h)$$ is the decoder output, and $$h = f(x)$$ is the encoder output.

It approximates the maximum likelihood training of a generative model with visible variables $$x$$ and hidden variables $$h$$, with an explicit joint distribution $$p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$$. The log-likelihood can be decomposed as:
$$log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$$

We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $$h$$. With this chosen $$h$$, we maximize:
$$log(p_{model}(h, x)) = log(p_{model}(h)) + log(p_{model}(x | h))$$

Expressing the log-prior as an absolute value penalty, we obtain:
$$\Omega (h) = \lambda \sum\limits_i |h_i|$$
$$- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$$

Sparse Autoencoders

Undercomplete Autoencoders are an attempt to solve the problem of encoding and decoding being learned so successfully that they circumvent the true goal of autoencoders, which is to learn which features of the data are important. 

"Undercomplete" means that for an autoencoder with basic structure
 input->code->output, with encoding function g and decoding function h, we constrain h to have a lower dimension than the input. Because of this reduced granularity, it is hoped that h will capture only the most "salient features" of the input data during training.

The learning process just involves minimizing the cost function L(x, g(f(x)). where L is a loss function which delivers a panalty for dissimilarity between x and f(x). (example: mean squared error).

Undercomplete Autoencoders

In the overcomplete case, the hidden code has a greater dimension than the input. 
The problem with this case is the linear encoder and linear decoder only learn to copy the input to the output. They do not learn anything about the data distribution.

Overcomplete Autoencoders

One way to regularize autoencoder is to add a penalty term. The loss function is thus $L(x, g(f(x))) + \Omega(h, x)$, where $\Omega(h, x) = \lambda \sum_{i}\Vert\nabla x h_i\Vert^2$. It is also called a contractive autoencoder. 

Regularizing Autoencoder

Making the autoencoder network deep has several benefits. 1. It reduces the computational cost of a single hidden layer. 2. It decreases the amount of data needed for training. 3. Deep autoencoder network yields better results. 

Autoencoder Depth

All autoencoder training procedures involve a compromise between two forces:
                                                                                                                                        
1. Learning a representation h of a training example x such that x can be approximately recovered from h through a decoder.
                                                                                                                                                                                                
2. Satisfying the constraint or regularization penalty. 

Neither force alone would be useful.
The two forces together are useful because they force the hidden representation to capture information about the structure of the data-generating distribution.
                                                                                                                               


Learning Manifolds Using Autoencoder

Some autoencoders explicitly represent a probability distribution and admit straightforward ancestral sampling. Most other kinds of autoencoders require MCMC sampling. 
There is also a general Markov chain that can sample from any denoising autoencoder.

Drawing Samples From Autoencoders

```python
# <1> Define the input to the encoder (the image).
encoder_input = Input(shape=self.input_dim, name='encoder_input')
x = encoder_input

for i in range(self.n_layers_encoder):
    conv_layer = Conv2D(
        filters=self.encoder_conv_filters[i],
        kernel_size=self.encoder_conv_kernel_size[i],
        strides=self.encoder_conv_strides[i],
        padding='same',
        name='encoder_conv_' + str(i)
    )
    # <2> Stack convolutional layers sequentially on top of each other.
    x = conv_layer(x)
    x = LeakyReLU()(x)
    if self.use_batch_norm:
        x = BatchNormalization()(x)
    if self.use_dropout:
        x = Dropout(rate=0.25)(x)

shape_before_flattening = K.int_shape(x)[1:]
x = Flatten()(x)

# <3> Dense layer that connects this vector to the 2D latent space.
encoder_output = Dense(self.z_dim, name='encoder_output')(x)

# <4> The Keras model that defines the encoder -- a model that takes an
# input image and encodes it into the 2D latent space.
self.encoder = Model(encoder_input, encoder_output)
```

Learn Before

Related

Learn After