Like the figure below, rows indicate examples within minibatches for the same training steps. Columns represent time step within the mean ﬁelds inference process. For each example, we sample a subset of the data variables (shaded black) to serve as inputs to the inference process; then run the mean ﬁeld inference process (arrows indicating the inﬂuence of variables). Dashed arrows indicate how the process could be unrolled for more steps. The grey nodes are the data variables that were not used as inputs to the inference process become targets.

Working Process of Multi-Prediction Deep Boltzmann Machine

Multi-prediction deep Boltzmann machine (MP-DBM) view the mean ﬁeld equations as deﬁning a family of recurrent networks to solve every possible inference problem. The model is trained to make each recurrent network obtain an accurate answer to the corresponding inference problem. 


State University of New York at Stony Brook

Software implementations of DBMs need to have many diﬀerent components for CD training of individual RBMs, PCD training of the full DBM, and training based on back-propagation through the MLP. The MLP on top of the Boltzmann machine loses many of the advantages of the Boltzmann machine probabilistic model, such as being able to perform inference when some input values are missing. 

There are two main ways to resolve the joint training problem of the deep Boltzmann machine, centered deep Boltzmann machine and multi-prediction deep Boltzmann machine


Jointly Training Deep Boltzmann Machines

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Centered deep Boltzmann machine reparametrizes the model to make the Hessian of the cost function better conditioned at the beginning of the learning process. This yields a model that can be trained without a greedy layer-wise pretraining stage.  The resulting model obtains excellent test set log-likelihood and produces high-quality samples. Unfortunately, it remains unable to compete with appropriately regularized MLPs as a classiﬁer.




Learn Before

Related

Learn After