Weight space symmetry serves as an example of how machine learning systems with hidden units are typically seen as non-identifiable.

By creating hidden units that simply swap both the input and output of two weight vectors, an equivalent model is created. If there are $m$ layers and $n$ swapping hidden units per layer, then there are $n!^m$ permutations of hidden units that create identical models.

Although this is a trivial example, it does show that hidden units alone can contribute greatly to a model's inherent non-identifiability.

Weight Space Symmetry

Hoyer et al. proved that the Additive Noise Model (ANM) is generally identifiable, demonstrating that if a joint distribution $$P_{X, Y}$$ satisfies an additive noise model with orientation $$X ightarrow Y$$, then $$P_{X, Y}$$ cannot satisfy an additive model with orientation $$Y ightarrow X$$. However, specific cases exist where ANM is non-identifiable; in particular, the linear Gaussian case is non-identifiable.

ANM Identifiability Result

A model is identifiable if some finite yet sufficiently large training set can rule out all model states except for one.

Thus if there are multiple states a model can not rule out, it is said to be non-identifiable.

University of Toledo

If failure is defined as a model becoming unable to escape a non-global local minima through training, then it is almost guaranteed that this will occur with a non-trivial and finite training set. This can be attributed to inherent $\textbf{model identifiability}$ issues which may allow uncountably infinite local minima to exist.

In practice, however, experts now believe that most relevant local minima are already close to the global minimum when given sufficiently large training sets. Although this is still an active area of research.

Likelihood that Gradient Descent will fail

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After