When applying a multi-layer perceptron (MLP) to two-dimensional images, both the inputs $\mathbf{X}$ and the hidden representations $\mathbf{H}$ can be treated as matrices with spatial structure. To allow every hidden unit to receive input from every pixel, the network's parameters are represented as a fourth-order weight tensor $\mathsf{W}$ and a bias matrix $\mathbf{U}$. The fully connected layer is formally expressed as:

$[\mathbf{H}]_{i, j} = [\mathbf{U}]_{i, j} + \sum_k \sum_l[\mathsf{W}]_{i, j, k, l} [\mathbf{X}]_{k, l} = [\mathbf{U}]_{i, j} + \sum_a \sum_b [\mathsf{V}]_{i, j, a, b} [\mathbf{X}]_{i+a, j+b}$

where $[\mathsf{V}]_{i, j, a, b} = [\mathsf{W}]_{i, j, i+a, j+b}$. A single fully connected layer mapping a $1000 \times 1000$ pixel image to a hidden representation of the same size using this parametrization requires $10^{12}$ parameters, which is computationally intractable.