When applying a multi-layer perceptron (MLP) to two-dimensional images, both the inputs $$\mathbf{X}$$ and the hidden representations $$\mathbf{H}$$ can be treated as matrices with spatial structure. To allow every hidden unit to receive input from every pixel, the network's parameters are represented as a fourth-order weight tensor $$\mathsf{W}$$ and a bias matrix $$\mathbf{U}$$. The fully connected layer is formally expressed as:

$$[\mathbf{H}]_{i, j} = [\mathbf{U}]_{i, j} + \sum_k \sum_l[\mathsf{W}]_{i, j, k, l} [\mathbf{X}]_{k, l} = [\mathbf{U}]_{i, j} + \sum_a \sum_b [\mathsf{V}]_{i, j, a, b} [\mathbf{X}]_{i+a, j+b}$$

where $$[\mathsf{V}]_{i, j, a, b} = [\mathsf{W}]_{i, j, i+a, j+b}$$. A single fully connected layer mapping a $$1000 \times 1000$$ pixel image to a hidden representation of the same size using this parametrization requires $$10^{12}$$ parameters, which is computationally intractable.

Constraining MLPs for Images

We can take the weights from the DBN and use them to define an MLP:
$$h^{(1)} = \sigma (b^{(1)} + v^{\top}W^{(1)})$$
$$h^{(1)} = \sigma (b^{(1)} + h^{(l-1)\top}W^{(l)})\forall l \in 2, ..., m$$
After initialization, we can train the MLP to perform a classification task, which is an example of discriminative fine-tuning.

University of Michigan - Ann Arbor

Training a Deep Belief Network (DBN) involves sequentially training Restricted Boltzmann Machines (RBMs). First, an RBM is trained to maximize $$\mathbb{E}_{v\sim p_{data}}\mathbf{log}p(v)$$ using contrastive divergence or stochastic maximum likelihood. Second, another RBM is trained to approximately maximize $$\mathbb{E}_{v\sim p_{data}}\mathbb{E}_{h^{(1)}\sim p^{(1)}(h^{(1)}|v)}\mathbf{log}p^{(2)}(h^{(1)})$$, where $$p^{(1)}$$ is the probability distribution represented by the first RBM and $$p^{(2)}$$ is the probability distribution represented by the second RBM.

Training Deep Belief Networks

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After