Zhang and Hyvärinen proved that the Post-NonLinear (PNL) model is generally identifiable. If a joint distribution $$P_{X, Y}$$ satisfies a PNL model with orientation $$X \rightarrow Y$$, then $$P_{X, Y}$$ cannot satisfy a PNL model with orientation $$Y \rightarrow X$$, except under specific conditions. The set of non-identifiable distributions $$P_{X, Y}$$ is larger for PNL than for the Additive Noise Model (ANM), but PNL is more general and handles a wider variety of observed distributions.

University of Michigan - Ann Arbor

Google

A model is identifiable if some finite yet sufficiently large training set can rule out all model states except for one.

Thus if there are multiple states a model can not rule out, it is said to be non-identifiable.

Model Identifiability

The Post-NonLinear (PNL) model is a generalization of the Additive Noise Model (ANM) that takes into account nonlinear interactions between the cause and the noise, as proposed by Zhang and Hyvärinen. A bivariate Post-NonLinear Model (PNL) $$X \rightarrow Y$$ is defined as $$Y := \hat{g}_Y (\hat{f}_Y(X) + N_Y)$$ with $$X \perp \! \! \! \perp N_Y$$, where $$\hat{f}_Y: \mathbb{R} \rightarrow \mathbb{R}$$ and $$\hat{g}_Y: \mathbb{R} \rightarrow \mathbb{R}$$ are Borel measurable functions, and $$\hat{g}_Y$$ is assumed to be invertible.

Post Nonlinear Model (PNL)

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Hoyer et al. proved that the Additive Noise Model (ANM) is generally identifiable, demonstrating that if a joint distribution $$P_{X, Y}$$ satisfies an additive noise model with orientation $$X ightarrow Y$$, then $$P_{X, Y}$$ cannot satisfy an additive model with orientation $$Y ightarrow X$$. However, specific cases exist where ANM is non-identifiable; in particular, the linear Gaussian case is non-identifiable.

ANM Identifiability Result

PNL Identifiability Result

A theoretical identifiability has been derived by Shimizu et al., who prove that if $$P_{X, Y}$$ admits the linear model $$Y := aX + N_Y$$ with $$X \perp \! \! \! \perp N_Y$$ (model 1), then there exist $$b \in \mathbb{R}$$ and a random variable $$N_X$$ such that $$X := bY + N_X$$ with $$Y \perp \! \! \! \perp N_X$$ (model 2) if and only if $$X$$ and $$N_Y$$ are Gaussian. In other words, there is only one non-identifiable case corresponding to the linear Gaussian case. Moreover, if $$X$$ or $$N_Y$$ is non-Gaussian, when the candidate linear model with orientation $$X \rightarrow Y$$ holds, the candidate linear model with orientation $$Y \rightarrow X$$ does not hold.

LiNGAM Identifiability Result

Weight space symmetry serves as an example of how machine learning models with hidden units are typically non-identifiable. By swapping the incoming and outgoing weights of two hidden units within the same layer, an equivalent model is produced. If there are $$m$$ layers and $$n$$ hidden units per layer, there are $$n!^m$$ permutations of hidden units that create identical models. This demonstrates that hidden units alone can contribute greatly to a model's inherent non-identifiability.

Weight Space Symmetry

The Post-NonLinear (PNL) model is evaluated using an independence score between the cause and the noise. As $$\hat{g}_Y$$ is assumed to be invertible, the noise variable can be recovered from the joint distribution $$P_{X, Y}$$ as: $$\hat{N}_Y = \hat{g}_Y^{-1}(Y) - \hat{f}_Y(X)$$. The noise variable is then estimated by functions $$l_1$$ and $$l_2$$ such as $$\hat{N}_Y = l_1(Y) - l_2(X)$$ with $$\hat{N}_Y$$ independent of $$X$$. This solves a constrained nonlinear Independent Component Analysis (ICA) problem, which can be achieved by minimizing $$I(X,\hat{N}_Y; \theta)$$, the mutual information between $$X$$ and $$\hat{N}_Y$$ with respect to the model parameter $$\theta$$. Symmetrically, an optimization of $$I(Y,\hat{N}_X; \theta)$$ is performed. The causal direction $$X \rightarrow Y$$ is preferred if $$I(X,\hat{N}_Y; \hat{\theta}) < I(Y,\hat{N}_X; \hat{\theta})$$, and $$Y \rightarrow X$$ otherwise.

Learn Before

Related