Due to the similarities between a Neural Autoregressive Density Estimator (NADE) and a Restricted Boltzmann Machine (RBM), one can perform mean field inferences on the NADE system in a computationally similar way. However, as the RBM is a recurrent network, only one inference can be made at a time. This is in contrast to the discrete nature of a NADE network, which allows it to perform a finite number of inferences $$k$$ simultaneously, where $$k$$ is less than the total number of inputs the NADE network was trained to handle.

NADE-k

The mean field approach is a commonly used distribution set that assumes $$q$$ is of the following form: $$q(h|v)=\prod_i q(h_i|v)$$ This specification for the factors of $$q$$ allows $$q(h|v)$$ to be optimized by optimizing a finite number of lower dimensional functions $$q(h_i|v)$$ using traditional optimization techniques. This is only one of many ways that the distribution $$q$$ can be limited. The technique is referred to as "structured variational inference" in general.

University of Toledo

Google

In variational inference and learning, inference is framed as an optimization problem where the evidence lower bound (ELBO), denoted as $$\mathcal{L}$$, is maximized with respect to an approximate distribution $$q$$, and learning is achieved by maximizing $$\mathcal{L}$$ with respect to the model parameters $$\theta$$. The equations for the ELBO are $$\mathcal{L}(v,\theta,q)= \log p(v;\theta) - D_{KL}(q(h|v)||p(h|v))$$ and $$\mathcal{L}(v,\theta,q)= \mathbb{E}_{h\sim q}(\log p(h,v))+H(q)$$. Because maximizing $$\mathcal{L}$$ with respect to $$\theta$$ without restrictions is often intractable, learning is performed over a restricted family of possible distributions for $$q$$. This restricted set is chosen to allow for the tractable computation of $$\mathbb{E}_{h\sim q}(\log p(h,v))$$.

Variational Inference and Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

When continuous latent variables are present in our graphical model, one can still use variational inference and learning by maximizing L, where L or L (v, θ, q) represents the evidence lower bound (ELBO) ; however, unlike with discrete latent variables we must now use calculus of variations.

When dealing with continuous latent variables the Mean Field Approximation:
	$$ q(\textbf{h}|\textbf{v}) = \prod_{i} q(h_{i}|\textbf{v}) $$ 
is fixed at $$q(h_{j}|\textbf{v}) $$ for all j =/= i making the optimal $$q(h_{i}|\textbf{v}) $$ is obtained by normalizing the following unnormalized distribution:
$$ \tilde{q}(h_{i}|\textbf{v})=exp(E_{h-i~ ~ q(h-i|v)} \log{\tilde{p}(\textbf{v},\textbf{h})})  $$
as long as p does not assign 0 probability to any joint conﬁguration of variables. 


Continuous Latent Variables in Variational Inference and Learning

Distribution q is defined - typically with a discrete lookup table - then its parameters are optimized, simply being a standard optimization problem.
Since we are optimizing in the inner loop of a learning algorithm, the operation must be very fast - typically special optimization algorithms are used, designed to solve comparatively small and simple problems in few iterations.

Learn Before

Related

Learn After