Weight space symmetry serves as an example of how machine learning models with hidden units are typically non-identifiable. By swapping the incoming and outgoing weights of two hidden units within the same layer, an equivalent model is produced. If there are $$m$$ layers and $$n$$ hidden units per layer, there are $$n!^m$$ permutations of hidden units that create identical models. This demonstrates that hidden units alone can contribute greatly to a model's inherent non-identifiability.

University of Toledo

Google

A model is identifiable if some finite yet sufficiently large training set can rule out all model states except for one.

Thus if there are multiple states a model can not rule out, it is said to be non-identifiable.

Model Identifiability

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Hoyer et al. proved that the Additive Noise Model (ANM) is generally identifiable, demonstrating that if a joint distribution $$P_{X, Y}$$ satisfies an additive noise model with orientation $$X ightarrow Y$$, then $$P_{X, Y}$$ cannot satisfy an additive model with orientation $$Y ightarrow X$$. However, specific cases exist where ANM is non-identifiable; in particular, the linear Gaussian case is non-identifiable.

ANM Identifiability Result

Zhang and Hyvärinen proved that the Post-NonLinear (PNL) model is generally identifiable. If a joint distribution $$P_{X, Y}$$ satisfies a PNL model with orientation $$X \rightarrow Y$$, then $$P_{X, Y}$$ cannot satisfy a PNL model with orientation $$Y \rightarrow X$$, except under specific conditions. The set of non-identifiable distributions $$P_{X, Y}$$ is larger for PNL than for the Additive Noise Model (ANM), but PNL is more general and handles a wider variety of observed distributions.

PNL Identifiability Result

A theoretical identifiability has been derived by Shimizu et al., who prove that if $$P_{X, Y}$$ admits the linear model $$Y := aX + N_Y$$ with $$X \perp \! \! \! \perp N_Y$$ (model 1), then there exist $$b \in \mathbb{R}$$ and a random variable $$N_X$$ such that $$X := bY + N_X$$ with $$Y \perp \! \! \! \perp N_X$$ (model 2) if and only if $$X$$ and $$N_Y$$ are Gaussian. In other words, there is only one non-identifiable case corresponding to the linear Gaussian case. Moreover, if $$X$$ or $$N_Y$$ is non-Gaussian, when the candidate linear model with orientation $$X \rightarrow Y$$ holds, the candidate linear model with orientation $$Y \rightarrow X$$ does not hold.

Learn Before

Related