The RNN-RBM sequence model is a generative model of a sequence of frames $$\boldsymbol x^{(t)}$$ consisting of an RNN that emits RBM parameters for each time step. This model differs from other approaches because it uses the RNN to emit all of the parameters of the RBM rather than just varying the bias parameters for each time step. The model is trained by back-propagating the gradient of the loss function through the RNN. The loss function is not applied directly to the RNN outputs, but it is applied to the RBM.

RNN-RBM Sequence Model

Boltzmann machines can be applied to sequence modelling tasks using conditional modelling. In sequence modelling, rather than mapping an input $$\boldsymbol x$$ to an output $$\boldsymbol y$$, the model must represent factors of the form $$p(\textbf{x}^{(\tau)}| \textbf{x}^{(1)}, \dotsc, \textbf{x}^{(\tau-1)})$$ to estimate a probability distribution over a sequence of variables, $$p(\textbf{x}^{(1)}, \dotsc, \textbf{x}^{(\tau)})$$.

New York University

Google

If a scenario requires the modelling of some output $$\boldsymbol y$$, where the different entries of $$\boldsymbol y$$ are related to each other and must obey certain constraints, from some input $$\boldsymbol x$$, we have what is called a structured output scenario. For example, $$\boldsymbol y$$ can be a waveform in a speech synthesis task which must sound coherent. The natural way to represent the relationships between the entries in $$\boldsymbol y$$ is to extend a Boltzmann machine to model conditional distributions of the form $$p(\mathbf{y}| \boldsymbol x)$$.

Boltzmann Machines for Structured Outputs

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After