Time-delay neural networks (TDNNs) are among the earliest examples of neural network architectures to exploit the concept of translation invariance. By applying a shared set of weights across different shifts in the input data, TDNNs significantly reduce the total number of parameters while maintaining the ability to detect patterns regardless of their spatial or temporal position.

Claude

The principle of translation invariance in computer vision asserts that a network should recognize objects regardless of their location in an image. When applied to constrain a multi-layer perceptron (MLP), this principle dictates that a shift in the input $$\mathbf{X}$$ must lead to an identical shift in the hidden representation $$\mathbf{H}$$. Consequently, the weight tensor $$\mathsf{V}$$ and bias $$\mathbf{U}$$ cannot depend on the absolute spatial coordinates $$(i, j)$$. Using a constant bias $$u$$ and a shared set of weights $$[\mathbf{V}]_{a, b}$$, the hidden representation simplifies to:

$$[\mathbf{H}]_{i, j} = u + \sum_a\sum_b [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}$$

This weight sharing dramatically reduces the parameter count (e.g., from $$10^{12}$$ to $$4 \times 10^6$$ for a 1-megapixel image) and effectively forms a convolution.

Translation Invariance

Dive into Deep Learning

Time-Delay Neural Networks (TDNNs)

The locality principle asserts that the earliest layers of a computer vision network should focus exclusively on local regions, without regard for the contents of the image in distant areas. Mathematically, this constrains the network's parameters such that outside a localized range ($$|a| > \Delta$$ or $$|b| > \Delta$$), the convolution weights are set to zero ($$[\mathbf{V}]_{a, b} = 0$$). The hidden representation is computed as:

$$[\mathbf{H}]_{i, j} = u + \sum_{a = -\Delta}^{\Delta} \sum_{b = -\Delta}^{\Delta} [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}$$

By restricting the network to glean relevant information only from a bounded window around each pixel, this principle drastically reduces the number of learnable parameters (e.g., from $$4 \times 10^6$$ down to $$4 \Delta^2$$).

Learn Before

Related