Weight space symmetry serves as an example of how machine learning systems with hidden units are typically seen as non-identifiable.

By creating hidden units that simply swap both the input and output of two weight vectors, an equivalent model is created. If there are $m$ layers and $n$ swapping hidden units per layer, then there are $n!^m$ permutations of hidden units that create identical models.

Although this is a trivial example, it does show that hidden units alone can contribute greatly to a model's inherent non-identifiability.