The probability distribution represented by a Deep Belief Network (DBN) is given by:
$$P(h^{(l)}, h^{(l-1)}) \propto \exp(b^{{(l)}^\top}h^{(l)} + b^{(l-1)\top}h^{(l-1)} + h^{(l-1)\top}W^{(l)}h^{(l)})$$
$$P(h_{i}^{(k)}=1 | h^{(k+1)}) = \sigma (b_{i}^{(k)}+W_{:,i}^{(k+1)\top}h^{(k+1)}) \forall i, \forall k \in 1, ..., l-2$$
$$P(v_i=1 | h^{(1)}) = \sigma (b_i^{(0)}+W_{:,i}^{(1)\top}h^{(1)}) \forall i$$
In the case of real-valued visible units, we substitute $$\mathbf{v}\sim\mathcal{N}(v;b^{(0)}+W^{(1)\top}h^{(1)},\beta^{-1})$$ with $$\beta$$ diagonal for tractability.

Google

Deep Belief Networks (DBNs) were one of the first nonconvolutional models to successfully admit training of deep architectures. They are generative models with several layers of latent or hidden variables in which the latent variables are typically binary, while the visible units may be binary or real. Every unit in each layer is connected to every unit in each neighboring layer.

Deep Belief Networks (DBNs)

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Training a Deep Belief Network (DBN) involves sequentially training Restricted Boltzmann Machines (RBMs). First, an RBM is trained to maximize $$\mathbb{E}_{v\sim p_{data}}\mathbf{log}p(v)$$ using contrastive divergence or stochastic maximum likelihood. Second, another RBM is trained to approximately maximize $$\mathbb{E}_{v\sim p_{data}}\mathbb{E}_{h^{(1)}\sim p^{(1)}(h^{(1)}|v)}\mathbf{log}p^{(2)}(h^{(1)})$$, where $$p^{(1)}$$ is the probability distribution represented by the first RBM and $$p^{(2)}$$ is the probability distribution represented by the second RBM.

Learn Before

Related