The mean field approach is a commonly used distribution set that assumes $q$ is of the following form:

$q(h|v)=\prod_i q(h_i|v)$

This specification for the factors of $q$ allows $q(h|v)$ to be optimized by optimizing a finite number of lower dimensional functions $q(h_i|v)$ using traditional optimization techniques.

This is only one of many ways that the distribution $q$ can be limited. The technique is referred to as "structured variational inference" in general.

Mean Field Approach

When continuous latent variables are present in our graphical model, one can still use variational inference and learning by maximizing L, where L or L (v, θ, q) represents the evidence lower bound (ELBO) ; however, unlike with discrete latent variables we must now use calculus of variations.

When dealing with continuous latent variables the Mean Field Approximation:
	$$ q(\textbf{h}|\textbf{v}) = \prod_{i} q(h_{i}|\textbf{v}) $$ 
is fixed at $$q(h_{j}|\textbf{v}) $$ for all j =/= i making the optimal $$q(h_{i}|\textbf{v}) $$ is obtained by normalizing the following unnormalized distribution:
$$ \tilde{q}(h_{i}|\textbf{v})=exp(E_{h-i~ ~ q(h-i|v)} \log{\tilde{p}(\textbf{v},\textbf{h})})  $$
as long as p does not assign 0 probability to any joint conﬁguration of variables. 


Continuous Latent Variables in Variational Inference and Learning

Distribution q is defined - typically with a discrete lookup table - then its parameters are optimized, simply being a standard optimization problem.
Since we are optimizing in the inner loop of a learning algorithm, the operation must be very fast - typically special optimization algorithms are used, designed to solve comparatively small and simple problems in few iterations.

Variational Inference with Discrete Latent Variables

Inferences can be made by maximizing $\mathcal{L}$ with respect to $q$ and learning can be achieved by maximizing $\mathcal{L}$ with respect to $\theta$ for the equation:

$\mathcal{L}(v,\theta,q)= logp(v;\theta) - D_{KL}(q(h|v)||p(h|v))$
$\mathcal{L}(v,\theta,q)= \mathbb{E}_{h\sim q}(logp(h,v))+H(q)$

However, the calculations needed to maximize $\mathcal{L}$ with respect to $\theta$ are often times intractible. Therefore, learning is performed on a restricted set of possible distributions for $q$.

This set should allow for the easy computation of $\mathbb{E}_qlogp(h,v)$ in order to remain tractible.

University of Toledo

List of techniques that help solve intractable inference problems in machine learning that usually arise from interactions between latent variables in a structured graphical model.

-Expectation Maximization

-MAP Inference and Sparse Coding

-Variational Inference and Learning

-Learned Approximate Inference



Approximate Inference 

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After