The authors note $θ_X$ and $θ_Y$ the vectors of parameters of the estimated mechanisms $\hat{f}_X$ and $\hat{f}_Y$. Note respectively $\theta_{N_X}$ and $\theta_{N_Y}$ the vectors of parameters of the distribution of the modeled noise variables $Q_{N_X}$ and $Q_{N_Y}$. The noise variables are independent. The global vector of parameters of the model is noted $\theta = (\theta_X, \theta_Y, \theta_{N_X}, \theta_{N_Y})$ . 

When $\mathcal{G} = X\rightarrow Y$, this candidate generative model (depicted on the figure) generates a distribution $Q_{X,Y}(\theta) = Q_X(\theta_X, \theta_{N_X})Q_{Y|X}(\theta_Y, \theta_{N_Y})$. 

When $\mathcal{G} =Y\rightarrow X$, this candidate model generates a distribution $Q_{X,Y}(\theta) = Q_Y(\theta_Y, \theta_{N_Y})Q_{X|Y}(\theta_X, \theta_{N_X})$. 

Structure of a Candidate Model

When modeling causal mechanisms within a candidate bivariate generative model, two primary characterizations must be specified:

1. The type of interaction between the noise variable and the cause.
2. The class of functions used to define the estimated mechanism $$\hat{f}$$.

Causal Mechanisms in a Candidate Generative Model

To model a continuous underlying bivariate generative process $$\mathcal{B}_{\mathcal{G}, f, P_N}$$ that is assumed to have generated observational data, a candidate bivariate generative model $$\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}$$ can be constructed. This candidate model is described by: - A structure defined by a causal orientation $$X \rightarrow Y$$ or $$Y \rightarrow X$$. - Estimated mechanisms modeled by $$\hat{f}$$. - An estimated distribution of noise $$Q_N$$.

University of Michigan - Ann Arbor

Google

The determination of the best explainable model is based in the literature on these main lines: 
 - Different candidate bivariate models (hypotheses) are evaluated in both directions.
 - For each candidate model one evaluates a score monitoring the trade-off between the fit of the model (meaning its adequacy to the observational data) and the complexity of the mechanisms involved. 
 - The model with the best score is returned, with its corresponding causal arrow X → Y or Y → X .

Determination of the Best Explainable Model

A bivariate structural model noted is a triplet $\mathcal{B}_{\mathcal{G}, f, P_N}$, where $\mathcal{G}$ is the causal graph X → Y or Y → X , $f = ( f_X , f_Y )$ is a couple of possibly nonlinear functions and $( N_X , N_Y )$ are two independent random variables drawn according to continuous distribution $P_N = (P_{N_X}, P_{N_Y})$, such that: 
 - $X := f_X ( N_X )$ and $Y := f_Y ( X , N_Y )$ if X → Y 
 - $Y := f_Y ( N_Y )$ and $X := f_X ( Y , N_X )$ if Y → X

Bivariate Structural Model

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

The authors introduce a term of complexity of a candidate bivariate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}(\theta)$ that we note $C_{\hat{B}}(\theta)$. This complexity notion refers to a simplicity prior on the underlying data generative process. It has been handled in different ways in the literature from fix a class of admissible mechanisms to more flexible criteria.

Th ecomplexity term can be defined through different approaches, including:
 - Explicit Class of Admissible Mechanisms
 - Kolmogorov Complexity

Complexity of the Model

Since we have defined the complexity $C_{\hat{\mathcal{B}}}(\theta)$ and the fit $S_{\hat{\mathcal{B}}}(\theta)$ of a candidate bivariate model parameterized by the vector of parameters θ . From a general point of view, we can now frame the model selection as a bi-objective trade-off with a Pareto front of optimal models.

This general bi-objective trade-off between fit and complexity is very general in science. It has been seen from two different angles. Some of the modeling approaches favor models of lower complexity, while others favor models with better explanatory power:
 - KISS Approach (Keep It Simple, Stupid)
 - KIDS Approach (Keep It Descriptive, Stupid)

Bi-objective Trade-Off between Fit and Complexity

Candidate Bivariate Generative Model

To evaluate the quality of a candidate generative model, a fit score $$S_{\hat{\mathcal{B}}}(\theta)$$ of a candidate model $$\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}(\theta)$$ is introduced. This score is implemented in various ways in the literature, but it always has the property of being minimal when $$P = Q$$ (perfect fit in the large sample limit). Examples of such scores include:
- Log-Likelihood Parametric Scores
- Implicit Fit Score Computed as an Independence Score Between Cause and Noise Variable
- Non-parametric Scores

Learn Before

Related

Learn After