Autoencoders try to train the network so that the weights for the encoder and decoder make the loss between the original data and its reconstruction as small as it is possible. 

What does the Autoencoder try to do ?

```python         
        #<1>The input to the autoencoder is the same as the input to the encoder.
        model_input = encoder_input
        #<2> The output from the autoencoder is the output from the encoder passed 
        # through the decoder.
        model_output = self.decoder(encoder_output)
        
        #<3> The Keras model that defines the full autoencoder—a model that takes 
        # an image, and passes it through the encoder and back out through the 
        # decoder to generate a reconstruction of the original image.
        self.model = Model(model_input, model_output)```

Putting it all together - AutoEncoder Code

1. Sampling for latent space can be very problematic
2. With autoencoders it is not guaranteed that we would obtain the same  quality when sampling from latent space
3. Some generated outputs are  poorly formed

Problems With Autoencoders

https://github.com/davidADSP/GDL_code/blob/master/models/AE.py

Reference to the AutoEncoder Code

__The decoder is a mirror image of the encoder, except instead of convolutional layers, we use convolutional transpose layers, as shown in the code. __

(Actually we don't have to write decoder as a mirror image of the encoder. But it is inevitable for the decoder to have exactly same size for the output image as we the input size of the encoder)
```python
     #<1> define the input to the decoder (the point in the latent space).
     decoder_input = Input(shape=(self.z_dim,), name='decoder_input')

        x = Dense(np.prod(shape_before_flattening))(decoder_input)
        x = Reshape(shape_before_flattening)(x)

        for i in range(self.n_layers_decoder):
            conv_t_layer = Conv2DTranspose(
                filters = self.decoder_conv_t_filters[i]
                , kernel_size = self.decoder_conv_t_kernel_size[i]
                , strides = self.decoder_conv_t_strides[i]
                , padding = 'same'
                , name = 'decoder_conv_t_' + str(i)
                )

            #<2>Stack convolutional transpose layers on top of each other.
            x = conv_t_layer(x)

            if i < self.n_layers_decoder - 1:
                x = LeakyReLU()(x)
                
                if self.use_batch_norm:
                    x = BatchNormalization()(x)
                
                if self.use_dropout:
                    x = Dropout(rate = 0.25)(x)
            else:
                x = Activation('sigmoid')(x)

        decoder_output = x
         
        #<3>The Keras model that defines the decoder—a model that takes a point in t 
        #the latent space and decodes it into the original image domain.
        self.decoder = Model(decoder_input, decoder_output)```

Autoencoder Decoder Code

```python
#<1>Define the input to the encoder (the image).
 encoder_input = Input(shape=self.input_dim, name='encoder_input')

        x = encoder_input

        for i in range(self.n_layers_encoder):
            conv_layer = Conv2D(
                filters = self.encoder_conv_filters[i]
                , kernel_size = self.encoder_conv_kernel_size[i]
                , strides = self.encoder_conv_strides[i]
                , padding = 'same'
                , name = 'encoder_conv_' + str(i)
                )

           #<2> stack convolutional layers sequentially on top of each other.
            x = conv_layer(x)

            x = LeakyReLU()(x)

            if self.use_batch_norm:
                x = BatchNormalization()(x)

            if self.use_dropout:
                x = Dropout(rate = 0.25)(x)

        shape_before_flattening = K.int_shape(x)[1:]
        
        x = Flatten()(x)
        #<3>dense layer that connects this vector to the 2D latent space.
        encoder_output= Dense(self.z_dim, name='encoder_output')(x)

        #<4> The Keras model that defines the encoder—a model that takes an input 
        # image and encodes it into the 2D latent space.
        self.encoder = Model(encoder_input, encoder_output)```


Autoencoder Encoder Sample Code

- Traditionally, autoencoders minimize some function $L(x,g(f(x)))$ where $L$ is a loss function penalizing $g(f(x))$ for being dissimilar from $x$, such as the $L^2$ norm of their difference. 
- A denoising auto encoder (DAE) instead minimizes $L(x,g(f(\widetilde{x})))$ where $\widetilde{x}$ is a copy of $x$ that has been corrupted by some form of noise. Denoising autoencoders must therefore undo this corruption rather than simply copying theirinput.
- Two assumptions are inherent to this approach: 1)Higher level representations are relatively stable and robust to the corruption of the input; 2) To perform denoising well, the model needs to extract features that capture useful structure in the input distribution.
- In other words, denoising is advocated as a training criterion for learning to extract useful features that will constitute better higher level representations of the input.

Denoising Autoencoders

- It is an autoencoder that trains with the reconstruction error involving a sparsity penalty $\Omega(h)$ on the code layer $h$: $L(x, g(f(x))) + \Omega(h)$, where $g(h)$ is the decoder output, and $h = f(x)$ the encoder output.
- It is a framework that approximates the maximum likelihood training of a generative model that has hidden layers.
- A model with visible variables $x$ and hidden variables $h$, with an explicit joint distribution $p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$. The log-likelihood can be decomposed as:
$$log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$$
                We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $h$, with this chosen $h$, we are maximizing
$$log(p_{model}(h, x)) = log(p_{model}(h) + log(p_{model}(x | h)$$
                Expressing the log-prior as an absolute value penalty, we obtain
$$\Omega (h) = \lambda \sum\limits_i |h_i|$$
$$- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$$

Sparse Autoencoders

Undercomplete Autoencoders are an attempt to solve the problem of encoding and decoding being learned so successfully that they circumvent the true goal of autoencoders, which is to learn which features of the data are important. 

"Undercomplete" means that for an autoencoder with basic structure
 input->code->output, with encoding function g and decoding function h, we constrain h to have a lower dimension than the input. Because of this reduced granularity, it is hoped that h will capture only the most "salient features" of the input data during training.

The learning process just involves minimizing the cost function L(x, g(f(x)). where L is a loss function which delivers a panalty for dissimilarity between x and f(x). (example: mean squared error).

Undercomplete Autoencoders

In the overcomplete case, the hidden code has a greater dimension than the input. 
The problem with this case is the linear encoder and linear decoder only learn to copy the input to the output. They do not learn anything about the data distribution.

Overcomplete Autoencoders

One way to regularize autoencoder is to add a penalty term. The loss function is thus $L(x, g(f(x))) + \Omega(h, x)$, where $\Omega(h, x) = \lambda \sum_{i}\Vert\nabla x h_i\Vert^2$. It is also called a contractive autoencoder. 

Regularizing Autoencoder

Making the autoencoder network deep has several benefits. 1. It reduces the computational cost of a single hidden layer. 2. It decreases the amount of data needed for training. 3. Deep autoencoder network yields better results. 

Autoencoder Depth

All autoencoder training procedures involve a compromise between two forces:
                                                                                                                                        
1. Learning a representation h of a training example x such that x can be approximately recovered from h through a decoder.
                                                                                                                                                                                                
2. Satisfying the constraint or regularization penalty. 

Neither force alone would be useful.
The two forces together are useful because they force the hidden representation to capture information about the structure of the data-generating distribution.
                                                                                                                               


Learning Manifolds Using Autoencoder

Some autoencoders explicitly represent a probability distribution and admit straightforward ancestral sampling. Most other kinds of autoencoders require MCMC sampling. 
There is also a general Markov chain that can sample from any denoising autoencoder.

Drawing Samples From Autoencoders

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


San Diego State University

Johns Hopkins University

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Autoencoders

In **autoencoder ** each image is simply mapped to one point in the latent space, while in **variational autoencoder** each image is mapped to multivariate normal distribution around that point.
Here is the image that demonstrates this difference:


AutoEncoder Encoder vs Variational AutoEncoder Encoder

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

Reference for Variational AutoEncoder Code

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Limitation(s) of Variational Autoencoder Framework


1. Establishes the most relevant high-level features.

2. Novel representations with high chance that after mapping will look like real 

Representational Learning - Strengths

From input text, meanings of the words were learnt automatically and useful representations were generated.

Representation Learning in NLP

A deep learning model is a powerful type of machine learning model differentiated by its architecture, which consists of many successive transformations of data chained together from top to bottom. This depth of continuous data transformations allows these models to tackle complex tasks by automatically learning multi-level representations. By doing so, deep learning effectively replaces not only the labor-intensive process of manual feature engineering, but also the shallow models that are typically used at the end of traditional machine learning pipelines.

Learn Before

Related

Learn After