A primary limitation is due to the examples used to train the classifiers. As widely known, the accuracy of trained classifiers depends on the quality of the training examples. In quite a few causal examples however, the joint distributions present typical features giving away the causality label, a phenomenon referred to as data leakage . 

Another issue would be the presence of biases in the training set of the classifiers. For example, if the causal pairs with one categorical variable and one numerical variable are always labelled such as categorical → numerical, the learning algorithms might learn biased features on distributions due to the training set. Such biases hinder the generality of the causality classifiers, as they might be exploited by learning algorithms and induce the biased hypotheses.

Limitation of Training Data Quality

A second limitation related with the data is their insufficient amount. As far as neural nets and deep learning are involved in the learning process, the quantity of examples also becomes essential. 

Given the comparatively few variable pairs for which the causality label is known from prior knowledge, many authors thus rely on data augmentation, generating new artificial examples from scratch or by perturbing the available examples. However, theoretical results require that causal classifiers be trained and evaluated on examples following the same Mother Distribution. 

As in all machine learning problem, the simplest setting is the i.i.d. setting in which training and test data are drawn from the same distribution. The same applies to the cause-effect pair problem: higher performance is attained when the pairs are drawn from the same mother distribution. 

Unfortunately, in many real world applications, one does not know from which “mother distribution” a new incoming pair to be classified is drawn and one does not have labeled examples of cause-effect pairs from the “mother distribution” of interest.

Limitation of Training Data Quantity

There are two primary limitations of the Mother Distribution framework where we cast the pairwise causal discovery problem as a supervised learning problem, regarding the quality and the quantity of the training data respectively.

University of Michigan - Ann Arbor

The authors can cast the cause-effect pair problem as a regular machine learning classification problem in which input-output pairs $( P_{Π = π} ( X , Y )$, G = g ) are drawn randomly and independently from a $\textbf{“mother distribution” }$ $P_M( P_{Π = π} ( X , Y ), G = g )$, then divided into training and test sets. 

The problem is to “classify” distributions $P_{Π = π} ( X , Y )$ into one of four classes $g ∈ \{ X → Y , X ← Y , X \leftrightarrow Y , X ⊥ Y \} $

Cast Cause-Effect Pair Problem as Machine Learning Classification Problem

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

The mother distribution is a distribution over join distributions of pairs of variables. The authors adopt the following definition of a mother distribution:

A mother distribution $P_M( P_{Π = π} ( X , Y ), G = g )$ is a distribution over pairs $( P_{Π = π} ( X , Y ), G = g )$, where $Π$ and $G$ are random variables:
 - $Π = π$ indexes distribution $P_{Π = π} ( X , Y )$ over random variables $X$ and $Y$ ; $π$ may possibly take an infinite number of values (e.g. parameters of a generative process)
 - $G = g$ labels causal relationship, $g ∈ \{ X → Y , X ← Y , X \leftrightarrow Y , X ⊥ Y \}$.

Mother Distribution

For practical reasons, it is convenient to recast the problem as a two-class classification problem separating X → Y from all other cases. 

We need to build a classifier (eventually using training data) to predict the ground truth of causal relationships as accurately as possible. The authors propose to construct a causation coefficient, which is a discriminant function in this two-class classification problem.

Recast as a Two-Class Classification Problem

The authors consider that pairs of variables X and Y can be in one of four types of relationships: 
 - X causes Y ( X → Y )
 - Y causes X ( X ← Y )
 - X and Y are dependent, but not in a causal relationship ( X :left_right_arrow: Y ) 
 - X and Y are independent ( X ⊥ Y ). 
Any pair of variables ( X , Y ) is associated with one and only one such relationships, called “ground truth” G . By definition, we have:
$G=[X\rightarrow Y] \Rightarrow \exists f \wedge \exists N_y s.t. Y:=f(X,N_y)$
$G=[X\leftarrow Y] \Rightarrow \exists f \wedge \exists N_x s.t. X:=f(Y,N_x)$
$G=[X\leftrightarrow Y] \Rightarrow \exists f,g \wedge \exists N_x, N_y, Z s.t. X:= f(Z,N_x)\wedge y:= f(Z,N_y)$
$G= X ⊥ Y$  X and Y are independent, no functional relationship

The authors do not necessarily assume that ( X ⊥ N y ) nor that ( Y ⊥ N x ), thus there can be additional latent confounders in the first two cases X → Y and X ← Y.

Definition of Pairwise Causal and Non-causal Relationship (in the book)

A distinctive characteristic of the causal pairwise classification problem compared to the traditional classification problem is the nature of the samples points. In a regular classification problem a sample is a vector representing the position of the example in the feature space $\mathbb{R}^d$, where $d$ represents the number of features. In our pairwise classification problem, a sample is an empirical distribution, a set of points $\{  x_i,y_i \}_{i=1}^{n_j}$. 
Therefore, a feature construction step is added between the data and the learning algorithm.

Learn Before

Related

Learn After