- The canonical RBM is an energy-based model with binary visible and hidden units. 
- Its energy function is $E(v,h) = -b^Tv - c^Th - v^TWh$, where b, c, and W are unconstrained, real valued, learnable parameters. 
- It is easy to take its derivatives: $\frac{\partial}{\partial W_{i,j}}E(v,h) = -v_ih_j$.
- There are no direct interactions between any two visible units or any two hidden units (restrictions).
- The restrictions on the RBM structure yield the nice properties $p(h|v)=\prod_ip(h_i|v)$ and $p(v|h) = \prod_ip(v_i|h)$.
- The individual conditionals are simple to compute. For the binary RBM we obtain: $P(h_i=1|v) = \rho(v^TW_{:,i} + b_i)$, and $P(h_i=1|v) =1 -  \rho(v^TW_{:,i} + b_i)$.
- Together, these properties allow for efficient block Gibbs sampling and efficient derivatives - make training convenient. Samples generated by Gibbs sampling from an RBM model are shown below:


Algorithm of Restricted Boltzmann Machine

- Restricted Boltzmann Machines are shallow, two-layer neural nets that constitute the building blocks of deep-belief networks. The first layer of the RBM is called the visible, or input layer, and the second is the hidden layer.
- Forward Pass: Each node of the visible layer is multiplied by a separate weight, the products are summed, added to a bias, and  the result is passed through an activation function to produce the node’s output (the hidden layer).
- Because inputs from all visible nodes are being passed to all hidden nodes, an RBM can be defined as a symmetrical bipartite graph.
- Reconstruction phase: the activations of hidden layer become the input in a backward pass. They are multiplied by the same weights, one per internode edge. The sum of those products is added to a visible-layer bias at each visible node, and the output of those operations is a reconstruction; i.e. an approximation of the original input. This can be represented by the following diagram:

Structure of Restricted Boltzmann Machine

- On its forward pass, an RBM uses inputs to make predictions about node activations.
- On its backward pass, when activations are fed in and reconstructions, or guesses about the original data, are spit out, an RBM is attempting to estimate the probability of inputs given activations of hidden layer, which are weighted with the same coefficients as those used on the forward pass. 
- Reconstruction is making guesses about the probability distribution of the original input; i.e. the values of many varied points at once. This is known as generative learning.
- To measure the distance between its estimated probability distribution and the ground-truth distribution of the input, RBMs use Kullback Leibler Divergence. 
- KL-Divergence measures the non-overlapping, or diverging, areas under the two curves, and an RBM’s optimization algorithm attempts to minimize those areas so that the shared weights, when multiplied by activations of hidden layer one, produce a close approximation of the original input. 

Probability Distribution and Generative Learning

- Contrastive Divergence (CD)
- Stochastic Maximum Likelihood (SML)
- Persistent Contrastive Divergence (PCD)
- Ratio matching

Training Models for RBM

Replacing matrix multiplication by discrete convolution with a small kernel is the standard way to solve these problems for inputs with translation-invariant spatial or temporal structure. This convolution approach works well when applied to RBMs. Lee et al. demonstrated that probabilistic max-pooling could be used to build convolutional deep Boltzmann machines. 


Convolutional Boltzmann Machines

It is an algorithm which is useful for dimensionality reduction, classification, regression, collaborative filtering, feature learning, and topic modeling.

Restricted Boltzmann Machines are shallow, two-layer neural nets that constitute the building blocks of deep-belief networks. The first layer of the RBM is called the visible, or input layer, and the second is the hidden layer.

The restriction in a Restricted Boltzmann Machine is that there is no intra-layer communication.

Restricted Boltzmann Machine Applications

Due to the similarities between a NADE network and an RBM, one can perform mean field inferences on the NADE system in a computationally similar way. However, as the RBM is a recurrent network, only one inference can be made at a time. This is in contrast to the discrete nature of a NADE network, which allows it to perform a finite number of inferences $k$ simultaneously, where $k$ is less than the total number of inputs the NADE network was trained to handle.

NADE-k

-	Restricted Boltzmann Machine is an undirected graphical model that plays a major role in Deep Learning Framework in recent times. It was initially introduced as Harmonium by Paul Smolensky in 1986 and it gained big popularity in recent years in the context of the Netflix Prize where Restricted Boltzmann Machines achieved state of the art performance in collaborative filtering and have beaten most of the competition.


University of Massachusetts at Amherst

University of Michigan - Ann Arbor

The deep learning approach is to figure out the minimum amount of information that is absolutely needed and find out how to get a reasonable approximation of this information in the shortest amount of time. 

The latent variables are designed differently in which the training algorithm can invent the concepts it needs to model a particular dataset freely and deal with complex problems and are reusable in more different contexts.


The Deep Learning Approach to Structured Probabilistic Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Restricted Boltzmann Machine

Deep Boltzmann Machine (DBM) is a deep generative model. Unlike Restricted Boltzmann Machine, DBM has multiple layers of latent variable. In each layer, each of the latent variable is mutually independent, and conditioned on their neighboring layers. This property allows us to use fixed-point equations and find the true optimal mean field expectation. The use of mean field simulates human brain connections and DBM is therefore used in neuroscientific phenomena. One undesired property of DBM is that sampling is difficult.

Deep Boltzmann Machine

Deep graphical models have large groups of units that are all connected to other groups of units. They tend to connect each visible unit $v_i$ to many hidden units $h_i$ for $h$ to provide a distributed representation of $v_i$

Learn Before

Related

Learn After