- Contrastive Divergence (CD)
- Stochastic Maximum Likelihood (SML)
- Persistent Contrastive Divergence (PCD)
- Ratio matching

University of Michigan - Ann Arbor

-	Restricted Boltzmann Machine is an undirected graphical model that plays a major role in Deep Learning Framework in recent times. It was initially introduced as Harmonium by Paul Smolensky in 1986 and it gained big popularity in recent years in the context of the Netflix Prize where Restricted Boltzmann Machines achieved state of the art performance in collaborative filtering and have beaten most of the competition.


Restricted Boltzmann Machine

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

- On the forward pass, an RBM uses input values to compute the hidden layer's activations (its predictions). - On the backward pass, those hidden activations are fed back through the same weights to produce a reconstruction: an estimate of $$P(\text{input} \mid \text{hidden activations})$$. - Because this reconstruction estimates a probability distribution over the input rather than a single deterministic value, the process is called generative learning. - An RBM compares its estimated distribution to the true input distribution using Kullback-Leibler (KL) Divergence, which measures the non-overlapping area between the two distributions. - Training minimizes this KL Divergence, adjusting the shared weights so that the reconstruction becomes an increasingly close approximation of the original input.

Probability Distribution and Generative Learning

Replacing matrix multiplication by discrete convolution with a small kernel is the standard way to solve these problems for inputs with translation-invariant spatial or temporal structure. This convolution approach works well when applied to RBMs. Lee et al. demonstrated that probabilistic max-pooling could be used to build convolutional deep Boltzmann machines. 


Convolutional Boltzmann Machines

It is an algorithm which is useful for dimensionality reduction, classification, regression, collaborative filtering, feature learning, and topic modeling.

Restricted Boltzmann Machines are shallow, two-layer neural nets that constitute the building blocks of deep-belief networks. The first layer of the RBM is called the visible, or input layer, and the second is the hidden layer.

The restriction in a Restricted Boltzmann Machine is that there is no intra-layer communication.

Restricted Boltzmann Machine Applications

Due to the similarities between a Neural Autoregressive Density Estimator (NADE) and a Restricted Boltzmann Machine (RBM), one can perform mean field inferences on the NADE system in a computationally similar way. However, as the RBM is a recurrent network, only one inference can be made at a time. This is in contrast to the discrete nature of a NADE network, which allows it to perform a finite number of inferences $$k$$ simultaneously, where $$k$$ is less than the total number of inputs the NADE network was trained to handle.

NADE-k

The canonical Restricted Boltzmann Machine (RBM) is an energy-based model with binary visible and hidden units. Its energy function is:

$$E(v,h) = -b^Tv - c^Th - v^TWh$$

where $$b$$, $$c$$, and $$W$$ are unconstrained, real-valued, learnable parameters. It is easy to take its derivatives:

$$\frac{\partial}{\partial W_{i,j}}E(v,h) = -v_ih_j$$

There are no direct interactions between any two visible units or any two hidden units (restrictions). The restrictions on the RBM structure yield the properties:

$$p(h|v)=\prod_ip(h_i|v)$$
$$p(v|h) = \prod_ip(v_i|h)$$

The individual conditionals are simple to compute. For the binary RBM we obtain:

$$P(h_i=1|v) = \rho(v^TW_{:,i} + b_i)$$
$$P(h_i=0|v) = 1 - \rho(v^TW_{:,i} + b_i)$$

Together, these properties allow for efficient block Gibbs sampling and efficient derivatives, making training convenient. Samples generated by Gibbs sampling from an RBM model are shown below:

Mathematical Formulation of Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are shallow, two-layer neural networks that constitute the building blocks of deep-belief networks. The first layer of the RBM is the visible (input) layer, and the second is the hidden layer. Because inputs from all visible nodes are passed to all hidden nodes, an RBM can be defined as a symmetrical bipartite graph.

During the forward pass, each node of the visible layer is multiplied by a separate weight, the products are summed, added to a bias, and the result is passed through an activation function to produce the hidden layer's output. In the reconstruction phase, the hidden layer's activations become the input in a backward pass. They are multiplied by the same weights, one per internode edge. The sum of those products is added to a visible-layer bias at each visible node, producing a reconstruction (an approximation of the original input). This can be represented by the following diagram:

Learn Before

Related

Related