Learn Before
Concept

Calculations for Inference Optimization

The bound is defined as: L(v,θ,q)=logp(v;θ)DKL(q(hv)p(hv;θ))\mathcal{L}(v,\theta,q)=log p(v;\theta)-D_{KL} (q(h|v)||p(h|v;\theta)), where qq is an arbitrary probability distribution over hh. L\mathcal{L} always has at most the same value as the desired log-probability, since the difference between logp(v)log p(v) and L(v,θ,q)\mathcal{L}(v,\theta,q) is given by the KL divergence, which is always nonnegative. The two are equal if and only if qq is the same distribution as p(hv)p(h|v). L\mathcal{L} can be rearranged through algebra into the simpler form L(v,θ,q)=Ehq[logp(h,v)]+H(q)\mathcal{L}(v,\theta,q)=\mathbb{E}_{h\sim q}[log p(h,v)]+H(q). Thus, we can think of inference as the procedure for finding the qq that maximizes L\mathcal{L}.

0

1

Updated 2021-07-29

References


Tags

Data Science