1Cademy - Sparse Autoencoders

How it works Courses Research Communities Benefits About Us

Learn Before

Autoencoders

Concept

Sparse Autoencoders

It is an autoencoder that trains with the reconstruction error involving a sparsity penalty $\Omega(h)$ on the code layer $h$ : $L(x, g(f(x))) + \Omega(h)$ , where $g(h)$ is the decoder output, and $h = f(x)$ the encoder output.
It is a framework that approximates the maximum likelihood training of a generative model that has hidden layers.
A model with visible variables $x$ and hidden variables $h$ , with an explicit joint distribution $p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$ . The log-likelihood can be decomposed as: $log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$ We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $h$ , with this chosen $h$ , we are maximizing $log(p_{model}(h, x)) = log(p_{model}(h) + log(p_{model}(x | h)$ Expressing the log-prior as an absolute value penalty, we obtain $\Omega (h) = \lambda \sum\limits_i |h_i|$ $- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$

0

1

Updated 2021-07-23

Contributors are:

Woongjin Jang

Who are from:

University of Michigan - Ann Arbor

University of Michigan - Ann Arbor

References

Tags

Data Science

Related

Learn After

Introduction to Autoencoders