Boltzmann machines can also be used for sequence modelling tasks using the same concept of conditional modeling. The difference lies in the fact that in sequence modeling rather than mapping an input $\boldsymbol x$ to an ouptut $\boldsymbol y$ , the model must represent factors of the form $p(\textbf{x}^{(\tau)}| \textbf{x}^{(1)}, \dotsc, \textbf{x}^{(\tau-1)})$ in order to estimate a probability distribution over a sequence of variables, $p(\textbf{x}^{(1)}, \dotsc,  \textbf{x}^{(\tau)})$

Boltzmann Machines for Sequence Modelling

If a scenario requires the modelling of some output $\boldsymbol y$, where the different entries of $\boldsymbol y$ are related to each other and must obey certain constraints, from some input $\boldsymbol x$, we have what is called a structured output scenario. For example, $\boldsymbol y$ can be a waveform in a speech synthesis task which must sound coherent. The natural way to represent the relationships between the entries in $\boldsymbol y$ is to extend a Boltzmann machine to model conditional distributions of the form $p(\mathbf{y}| \boldsymbol x)$.

New York University

- Restricted Boltzmann Machines are shallow, two-layer neural nets that constitute the building blocks of deep-belief networks. The first layer of the RBM is called the visible, or input layer, and the second is the hidden layer.
- Forward Pass: Each node of the visible layer is multiplied by a separate weight, the products are summed, added to a bias, and  the result is passed through an activation function to produce the node’s output (the hidden layer).
- Because inputs from all visible nodes are being passed to all hidden nodes, an RBM can be defined as a symmetrical bipartite graph.
- Reconstruction phase: the activations of hidden layer become the input in a backward pass. They are multiplied by the same weights, one per internode edge. The sum of those products is added to a visible-layer bias at each visible node, and the output of those operations is a reconstruction; i.e. an approximation of the original input. This can be represented by the following diagram:

Structure of Restricted Boltzmann Machine

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After