A first type of methods in the literature defines a rudimentary notion of complexity: all mechanisms $\hat{f}$ belonging to a particular class $\mathcal{F}$ of bivariate FCM are assumed to be simple, while the others are assumed to be complex.

For these methods, we write $C_{\hat{B}}(\theta) = 0$ if $\hat{f} \in \mathcal{F}$ and $C_{\hat{B}}(\theta) = 1$ otherwise.

University of Michigan - Ann Arbor

The authors introduce a term of complexity of a candidate bivariate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}(\theta)$ that we note $C_{\hat{B}}(\theta)$. This complexity notion refers to a simplicity prior on the underlying data generative process. It has been handled in different ways in the literature from fix a class of admissible mechanisms to more flexible criteria.

Th ecomplexity term can be defined through different approaches, including:
 - Explicit Class of Admissible Mechanisms
 - Kolmogorov Complexity

Complexity of the Model

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Explicit Class of Admissible Mechanisms

In this approach, the basic postulate that “the factorization of the joint density function $P_{cause,effect}$ into $P_{cause} P_{effect|cause}$ should lead to a simpler model than $P_{effect} P_{cause|effect}$ ”, can be expressed with the Kolmogorov complexity framework:
$K(P_{cause}) + K(P_{effect|cause}) \overset{+}\leq K(P_{effect}) + K(P_{cause|effect}) $

This inequality comes from the postulate of algorithmic independence between the distribution of the cause $P_{cause}$ and the distribution of the causal mechanism $P_{effect|cause}$ stated by Janzing and Schölkopf as: 
$I(P_{cause} : P_{effect|cause}) \overset{+}=0$
where $I(P_{cause} : P_{effect|cause})$ denotes algorithmic mutual information. 

Kolmogorov complexity and algorithmic mutual information are not computable in practice but they have led to two different practical implementations in the literature:
 - Model Selection with Minimum Message Length Principle (MML)
 - Independence Between Cause and Mechanism

Learn Before

Related