One way to regularize autoencoder is to add a penalty term. The loss function is thus $L(x, g(f(x))) + \Omega(h, x)$, where $\Omega(h, x) = \lambda \sum_{i}\Vert\nabla x h_i\Vert^2$. It is also called a contractive autoencoder. 

Johns Hopkins University

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Autoencoders try to train the network so that the weights for the encoder and decoder make the loss between the original data and its reconstruction as small as it is possible. 

What does the Autoencoder try to do ?

```python         
        #<1>The input to the autoencoder is the same as the input to the encoder.
        model_input = encoder_input
        #<2> The output from the autoencoder is the output from the encoder passed 
        # through the decoder.
        model_output = self.decoder(encoder_output)
        
        #<3> The Keras model that defines the full autoencoder—a model that takes 
        # an image, and passes it through the encoder and back out through the 
        # decoder to generate a reconstruction of the original image.
        self.model = Model(model_input, model_output)```

Putting it all together - AutoEncoder Code

1. Sampling for latent space can be very problematic
2. With autoencoders it is not guaranteed that we would obtain the same  quality when sampling from latent space
3. Some generated outputs are  poorly formed

Problems With Autoencoders

https://github.com/davidADSP/GDL_code/blob/master/models/AE.py

Reference to the AutoEncoder Code

__The decoder is a mirror image of the encoder, except instead of convolutional layers, we use convolutional transpose layers, as shown in the code. __

(Actually we don't have to write decoder as a mirror image of the encoder. But it is inevitable for the decoder to have exactly same size for the output image as we the input size of the encoder)
```python
     #<1> define the input to the decoder (the point in the latent space).
     decoder_input = Input(shape=(self.z_dim,), name='decoder_input')

        x = Dense(np.prod(shape_before_flattening))(decoder_input)
        x = Reshape(shape_before_flattening)(x)

        for i in range(self.n_layers_decoder):
            conv_t_layer = Conv2DTranspose(
                filters = self.decoder_conv_t_filters[i]
                , kernel_size = self.decoder_conv_t_kernel_size[i]
                , strides = self.decoder_conv_t_strides[i]
                , padding = 'same'
                , name = 'decoder_conv_t_' + str(i)
                )

            #<2>Stack convolutional transpose layers on top of each other.
            x = conv_t_layer(x)

            if i < self.n_layers_decoder - 1:
                x = LeakyReLU()(x)
                
                if self.use_batch_norm:
                    x = BatchNormalization()(x)
                
                if self.use_dropout:
                    x = Dropout(rate = 0.25)(x)
            else:
                x = Activation('sigmoid')(x)

        decoder_output = x
         
        #<3>The Keras model that defines the decoder—a model that takes a point in t 
        #the latent space and decodes it into the original image domain.
        self.decoder = Model(decoder_input, decoder_output)```

Autoencoder Decoder Code

```python
#<1>Define the input to the encoder (the image).
 encoder_input = Input(shape=self.input_dim, name='encoder_input')

        x = encoder_input

        for i in range(self.n_layers_encoder):
            conv_layer = Conv2D(
                filters = self.encoder_conv_filters[i]
                , kernel_size = self.encoder_conv_kernel_size[i]
                , strides = self.encoder_conv_strides[i]
                , padding = 'same'
                , name = 'encoder_conv_' + str(i)
                )

           #<2> stack convolutional layers sequentially on top of each other.
            x = conv_layer(x)

            x = LeakyReLU()(x)

            if self.use_batch_norm:
                x = BatchNormalization()(x)

            if self.use_dropout:
                x = Dropout(rate = 0.25)(x)

        shape_before_flattening = K.int_shape(x)[1:]
        
        x = Flatten()(x)
        #<3>dense layer that connects this vector to the 2D latent space.
        encoder_output= Dense(self.z_dim, name='encoder_output')(x)

        #<4> The Keras model that defines the encoder—a model that takes an input 
        # image and encodes it into the 2D latent space.
        self.encoder = Model(encoder_input, encoder_output)```


Autoencoder Encoder Sample Code

- Traditionally, autoencoders minimize some function $L(x,g(f(x)))$ where $L$ is a loss function penalizing $g(f(x))$ for being dissimilar from $x$, such as the $L^2$ norm of their difference. 
- A denoising auto encoder (DAE) instead minimizes $L(x,g(f(\widetilde{x})))$ where $\widetilde{x}$ is a copy of $x$ that has been corrupted by some form of noise. Denoising autoencoders must therefore undo this corruption rather than simply copying theirinput.
- Two assumptions are inherent to this approach: 1)Higher level representations are relatively stable and robust to the corruption of the input; 2) To perform denoising well, the model needs to extract features that capture useful structure in the input distribution.
- In other words, denoising is advocated as a training criterion for learning to extract useful features that will constitute better higher level representations of the input.

Denoising Autoencoders

- It is an autoencoder that trains with the reconstruction error involving a sparsity penalty $\Omega(h)$ on the code layer $h$: $L(x, g(f(x))) + \Omega(h)$, where $g(h)$ is the decoder output, and $h = f(x)$ the encoder output.
- It is a framework that approximates the maximum likelihood training of a generative model that has hidden layers.
- A model with visible variables $x$ and hidden variables $h$, with an explicit joint distribution $p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$. The log-likelihood can be decomposed as:
$$log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$$
                We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $h$, with this chosen $h$, we are maximizing
$$log(p_{model}(h, x)) = log(p_{model}(h) + log(p_{model}(x | h)$$
                Expressing the log-prior as an absolute value penalty, we obtain
$$\Omega (h) = \lambda \sum\limits_i |h_i|$$
$$- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$$

Sparse Autoencoders

Undercomplete Autoencoders are an attempt to solve the problem of encoding and decoding being learned so successfully that they circumvent the true goal of autoencoders, which is to learn which features of the data are important. 

"Undercomplete" means that for an autoencoder with basic structure
 input->code->output, with encoding function g and decoding function h, we constrain h to have a lower dimension than the input. Because of this reduced granularity, it is hoped that h will capture only the most "salient features" of the input data during training.

The learning process just involves minimizing the cost function L(x, g(f(x)). where L is a loss function which delivers a panalty for dissimilarity between x and f(x). (example: mean squared error).

Undercomplete Autoencoders

In the overcomplete case, the hidden code has a greater dimension than the input. 
The problem with this case is the linear encoder and linear decoder only learn to copy the input to the output. They do not learn anything about the data distribution.

Overcomplete Autoencoders

Regularizing Autoencoder

Making the autoencoder network deep has several benefits. 1. It reduces the computational cost of a single hidden layer. 2. It decreases the amount of data needed for training. 3. Deep autoencoder network yields better results. 

Autoencoder Depth

All autoencoder training procedures involve a compromise between two forces:
                                                                                                                                        
1. Learning a representation h of a training example x such that x can be approximately recovered from h through a decoder.
                                                                                                                                                                                                
2. Satisfying the constraint or regularization penalty. 

Neither force alone would be useful.
The two forces together are useful because they force the hidden representation to capture information about the structure of the data-generating distribution.
                                                                                                                               


Learning Manifolds Using Autoencoder

Some autoencoders explicitly represent a probability distribution and admit straightforward ancestral sampling. Most other kinds of autoencoders require MCMC sampling. 
There is also a general Markov chain that can sample from any denoising autoencoder.

Learn Before

Related