Distribution q is defined - typically with a discrete lookup table - then its parameters are optimized, simply being a standard optimization problem.
Since we are optimizing in the inner loop of a learning algorithm, the operation must be very fast - typically special optimization algorithms are used, designed to solve comparatively small and simple problems in few iterations.

University of California, Santa Cruz

Inferences can be made by maximizing $\mathcal{L}$ with respect to $q$ and learning can be achieved by maximizing $\mathcal{L}$ with respect to $\theta$ for the equation:

$\mathcal{L}(v,\theta,q)= logp(v;\theta) - D_{KL}(q(h|v)||p(h|v))$
$\mathcal{L}(v,\theta,q)= \mathbb{E}_{h\sim q}(logp(h,v))+H(q)$

However, the calculations needed to maximize $\mathcal{L}$ with respect to $\theta$ are often times intractible. Therefore, learning is performed on a restricted set of possible distributions for $q$.

This set should allow for the easy computation of $\mathbb{E}_qlogp(h,v)$ in order to remain tractible.

Variational Inference and Learning

The mean field approach is a commonly used distribution set that assumes $q$ is of the following form:

$q(h|v)=\prod_i q(h_i|v)$

This specification for the factors of $q$ allows $q(h|v)$ to be optimized by optimizing a finite number of lower dimensional functions $q(h_i|v)$ using traditional optimization techniques.

This is only one of many ways that the distribution $q$ can be limited. The technique is referred to as "structured variational inference" in general.

Mean Field Approach

When continuous latent variables are present in our graphical model, one can still use variational inference and learning by maximizing L, where L or L (v, θ, q) represents the evidence lower bound (ELBO) ; however, unlike with discrete latent variables we must now use calculus of variations.

When dealing with continuous latent variables the Mean Field Approximation:
	$$ q(\textbf{h}|\textbf{v}) = \prod_{i} q(h_{i}|\textbf{v}) $$ 
is fixed at $$q(h_{j}|\textbf{v}) $$ for all j =/= i making the optimal $$q(h_{i}|\textbf{v}) $$ is obtained by normalizing the following unnormalized distribution:
$$ \tilde{q}(h_{i}|\textbf{v})=exp(E_{h-i~ ~ q(h-i|v)} \log{\tilde{p}(\textbf{v},\textbf{h})})  $$
as long as p does not assign 0 probability to any joint conﬁguration of variables. 


Learn Before

Related