A sparse autoencoder is an autoencoder whose training criterion involves a sparsity penalty $\Omega(h)$ on the code layer $h$. The loss function is $L(x, g(f(x))) + \Omega(h)$, where $g(h)$ is the decoder output, and $h = f(x)$ is the encoder output.

It approximates the maximum likelihood training of a generative model with visible variables $x$ and hidden variables $h$, with an explicit joint distribution p_{model}(x, h) = p_{model}(h)p_{model}(x | h). The log-likelihood can be decomposed as: $log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$

We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $h$. With this chosen $h$, we maximize: log(p_{model}(h, x)) = log(p_{model}(h)) + log(p_{model}(x | h))

Expressing the log-prior as an absolute value penalty, we obtain: $\Omega (h) = \lambda \sum\limits_i |h_i|$ $- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$