When the latent variables $$h$$ are discrete, the approximate posterior $$q$$ can be defined directly, typically as a discrete lookup table storing one probability value per state. Optimizing the parameters of this lookup table is then a standard optimization problem, unlike the continuous case, which requires calculus of variations. Because this optimization runs in the inner loop of a larger learning algorithm, it must be very fast, so specialized optimization algorithms designed to solve small, simple problems in very few iterations are typically used.

University of California, Santa Cruz

Google

In variational inference and learning, inference is framed as an optimization problem where the evidence lower bound (ELBO), denoted as $$\mathcal{L}$$, is maximized with respect to an approximate distribution $$q$$, and learning is achieved by maximizing $$\mathcal{L}$$ with respect to the model parameters $$\theta$$. The equations for the ELBO are $$\mathcal{L}(v,\theta,q)= \log p(v;\theta) - D_{KL}(q(h|v)||p(h|v))$$ and $$\mathcal{L}(v,\theta,q)= \mathbb{E}_{h\sim q}(\log p(h,v))+H(q)$$. Because maximizing $$\mathcal{L}$$ with respect to $$\theta$$ without restrictions is often intractable, learning is performed over a restricted family of possible distributions for $$q$$. This restricted set is chosen to allow for the tractable computation of $$\mathbb{E}_{h\sim q}(\log p(h,v))$$.

Variational Inference and Learning

When continuous latent variables are present in our graphical model, one can still use variational inference and learning by maximizing L, where L or L (v, θ, q) represents the evidence lower bound (ELBO) ; however, unlike with discrete latent variables we must now use calculus of variations.

When dealing with continuous latent variables the Mean Field Approximation:
	$$ q(\textbf{h}|\textbf{v}) = \prod_{i} q(h_{i}|\textbf{v}) $$ 
is fixed at $$q(h_{j}|\textbf{v}) $$ for all j =/= i making the optimal $$q(h_{i}|\textbf{v}) $$ is obtained by normalizing the following unnormalized distribution:
$$ \tilde{q}(h_{i}|\textbf{v})=exp(E_{h-i~ ~ q(h-i|v)} \log{\tilde{p}(\textbf{v},\textbf{h})})  $$
as long as p does not assign 0 probability to any joint conﬁguration of variables. 


Continuous Latent Variables in Variational Inference and Learning

Variational Inference with Discrete Latent Variables

The mean field approach is a commonly used distribution set that assumes $$q$$ is of the following form: $$q(h|v)=\prod_i q(h_i|v)$$ This specification for the factors of $$q$$ allows $$q(h|v)$$ to be optimized by optimizing a finite number of lower dimensional functions $$q(h_i|v)$$ using traditional optimization techniques. This is only one of many ways that the distribution $$q$$ can be limited. The technique is referred to as "structured variational inference" in general.

Learn Before

Related