Due to the similarities between a NADE network and an RBM, one can perform mean field inferences on the NADE system in a computationally similar way. However, as the RBM is a recurrent network, only one inference can be made at a time. This is in contrast to the discrete nature of a NADE network, which allows it to perform a finite number of inferences $k$ simultaneously, where $k$ is less than the total number of inputs the NADE network was trained to handle.

NADE-k

The mean field approach is a commonly used distribution set that assumes $q$ is of the following form:

$q(h|v)=\prod_i q(h_i|v)$

This specification for the factors of $q$ allows $q(h|v)$ to be optimized by optimizing a finite number of lower dimensional functions $q(h_i|v)$ using traditional optimization techniques.

This is only one of many ways that the distribution $q$ can be limited. The technique is referred to as "structured variational inference" in general.

University of Toledo

Inferences can be made by maximizing $\mathcal{L}$ with respect to $q$ and learning can be achieved by maximizing $\mathcal{L}$ with respect to $\theta$ for the equation:

$\mathcal{L}(v,\theta,q)= logp(v;\theta) - D_{KL}(q(h|v)||p(h|v))$
$\mathcal{L}(v,\theta,q)= \mathbb{E}_{h\sim q}(logp(h,v))+H(q)$

However, the calculations needed to maximize $\mathcal{L}$ with respect to $\theta$ are often times intractible. Therefore, learning is performed on a restricted set of possible distributions for $q$.

This set should allow for the easy computation of $\mathbb{E}_qlogp(h,v)$ in order to remain tractible.

Variational Inference and Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Mean Field Approach

When continuous latent variables are present in our graphical model, one can still use variational inference and learning by maximizing L, where L or L (v, θ, q) represents the evidence lower bound (ELBO) ; however, unlike with discrete latent variables we must now use calculus of variations.

When dealing with continuous latent variables the Mean Field Approximation:
	$$ q(\textbf{h}|\textbf{v}) = \prod_{i} q(h_{i}|\textbf{v}) $$ 
is fixed at $$q(h_{j}|\textbf{v}) $$ for all j =/= i making the optimal $$q(h_{i}|\textbf{v}) $$ is obtained by normalizing the following unnormalized distribution:
$$ \tilde{q}(h_{i}|\textbf{v})=exp(E_{h-i~ ~ q(h-i|v)} \log{\tilde{p}(\textbf{v},\textbf{h})})  $$
as long as p does not assign 0 probability to any joint conﬁguration of variables. 


Continuous Latent Variables in Variational Inference and Learning

Distribution q is defined - typically with a discrete lookup table - then its parameters are optimized, simply being a standard optimization problem.
Since we are optimizing in the inner loop of a learning algorithm, the operation must be very fast - typically special optimization algorithms are used, designed to solve comparatively small and simple problems in few iterations.

Learn Before

Related

Learn After