https://www.jeremyjordan.me/autoencoders/

University of Michigan - Ann Arbor

- It is an autoencoder that trains with the reconstruction error involving a sparsity penalty $\Omega(h)$ on the code layer $h$: $L(x, g(f(x))) + \Omega(h)$, where $g(h)$ is the decoder output, and $h = f(x)$ the encoder output.
- It is a framework that approximates the maximum likelihood training of a generative model that has hidden layers.
- A model with visible variables $x$ and hidden variables $h$, with an explicit joint distribution $p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$. The log-likelihood can be decomposed as:
$$log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$$
                We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $h$, with this chosen $h$, we are maximizing
$$log(p_{model}(h, x)) = log(p_{model}(h) + log(p_{model}(x | h)$$
                Expressing the log-prior as an absolute value penalty, we obtain
$$\Omega (h) = \lambda \sum\limits_i |h_i|$$
$$- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$$

Learn Before

Related