Indeed in the literature, we distinguish mainly additive noise interaction of the form $y:=\hat{f}_Y(X) + N_Y$, or complex noise interaction of the form $Y:= \hat{f}_Y(X,N_Y)$, where the cause variable and the noise are mixed with a non-linear mechanism.”


Type of Interaction between the Noise Variable and the Cause

It may range from linear mechanisms as in LiNGAM algorithm to complex non-linear mechanisms modeled with Gaussian processes or neural networks. 

In general, the more complex the mechanisms are, the more the candidate model can fit the data and the more the model is general (meaning that it can be applied to a wide variety of cases). However, it may result in more difficulty to assess the causal orientation because the candidate model has more chances to fit equally well the data in both directions. 

The Class of Functions Used to Define $\hat{f}$

Now we have all the ingredients required to sample data points of the estimated distribution $Q_{X , Y}$ with the generative model by proceeding as follow: 

1. Draw $\{(n_{X,j}, n_{Y,j})\}_{j=1}^n$ , n samples independent and identically distributed from the joint distribution $Q_{N_X}(\theta_{N_X})\times Q_{N_Y}(\theta_{N_Y})$ of independent noise variables $( N_X , N_Y )$. 
2. Generate n samples $\hat{\mathcal{D}} = \{(\hat{x}_j, \hat{y}_j) \}_{j=1}^n$, where each estimated sample $\hat{x}_j$ of variable $X$ is computed from $\hat{f}_X(\theta_X)$ with the j-th estimated samples $n_{X , j}$; then each estimated sample $\hat{y}_j$ of variable $Y$ is computed from $\hat{f}_Y(\theta_Y)$ with the j-th estimated samples $n_{Y , j}$ and $\hat{x}_j$. 

Generative candidate models supports both interventions and counterfactual reasoning.

Sampling Data Points with a Candidate Model

After having defined the overall structure, one needs to model the mechanisms, there are two characterizations that need to be specified:
 - Type of interaction between the noise variable and the cause.
 - The class of functions used to define $\hat{f}$.

University of Michigan - Ann Arbor

In order to model such continuous underlying bivariate generative process $\mathcal{B}_{\mathcal{G}, f, P_N}$ which is assumed to have generated the data, the author introduce the notion of candidate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}$ described by: 
 - a structure defined by a causal orientation X → Y or Y → X 
 - its estimated mechanisms modeled by $\hat{f}$
 -  an estimated distribution of noise $Q_N$ 

Candidate Bivariate Models

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

The authors note $θ_X$ and $θ_Y$ the vectors of parameters of the estimated mechanisms $\hat{f}_X$ and $\hat{f}_Y$. Note respectively $\theta_{N_X}$ and $\theta_{N_Y}$ the vectors of parameters of the distribution of the modeled noise variables $Q_{N_X}$ and $Q_{N_Y}$. The noise variables are independent. The global vector of parameters of the model is noted $\theta = (\theta_X, \theta_Y, \theta_{N_X}, \theta_{N_Y})$ . 

When $\mathcal{G} = X\rightarrow Y$, this candidate generative model (depicted on the figure) generates a distribution $Q_{X,Y}(\theta) = Q_X(\theta_X, \theta_{N_X})Q_{Y|X}(\theta_Y, \theta_{N_Y})$. 

When $\mathcal{G} =Y\rightarrow X$, this candidate model generates a distribution $Q_{X,Y}(\theta) = Q_Y(\theta_Y, \theta_{N_Y})Q_{X|Y}(\theta_X, \theta_{N_X})$. 

Learn Before

Related

Learn After