When it comes to the methods of evaluation of cause-effect pair algorithms,  the algorithm developers want to characterize their new algorithms both theoretically and empirically, so they care about:
 - backing algorithms with theoretical justifications (identifiability)
 - empirical assessment methods

Cause-Effect Pair Algorithm Evaluation Methods (for Algorithm Developers)

For practitioners who want to use causation coefficients in various real-world applications (in medicine or epidemiology, social sciences, econometrics, etc.), “evaluation”means assessing the reliability of given putative cause-effect relationship “discoveries” inferred from observational data. They do not want to evaluate methods, they want to use them to identify potential cause-effect relationships in pairs of real variables.

If practitioners are provided with a causation coefficient, they have expectation on:
 - Prerequisites
 - Efficacy
 - Post-hoc Evaluation

Cause-Effect Pair Algorithm Evaluation Methods (for Practitioners)

Benchmark organizers aim at organizing a large scale evaluation of cause-effect pair methods, so they make efforts on creating good benchmark datasets.

The options to get benchmark data include:
 - Facilitate the collection of real data
 - Create synthetic data
 - Generate artificial data

Cause-Effect Pair Algorithm Evaluation Methods (for Benchmark Organizers)

While talking about methods of evaluation of cause-effect pair algorithms, there are three points of view : 
 - that of algorithm developers in need to characterize their new algorithms both theoretically and empirically
 - that of practitioners who want to ensure the validity of a potential discovery
 - that of benchmark organizers who want to make fair comparisons.

University of Michigan - Ann Arbor

For practical reasons, it is convenient to recast the problem as a two-class classification problem separating X → Y from all other cases. 

We need to build a classifier (eventually using training data) to predict the ground truth of causal relationships as accurately as possible. The authors propose to construct a causation coefficient, which is a discriminant function in this two-class classification problem.

Recast as a Two-Class Classification Problem

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Consider a pair of random variables $( X , Y )$ with ground truth $G ∈\{ X → Y , X ← Y , X \leftrightarrow Y , X ⊥ Y \} $, drawn from an unknown mother distribution. We call an estimator $\hat{G}$ of $G$ based on $( X , Y )$. 

The authors define a causation coefficient as follows:
A causation coefficient $C ( X , Y )$ is a real scalar value, such that the larger $C ( X , Y )$, the more confident we are that $G = X → Y$ . A causation coefficient must have the following desired properties:
1. [Anti-symmetry] 
$C(X,Y) = -C(Y,X)$
2. [Discriminant] 
$C(X,Y)>\theta \leq 0 \Rightarrow \hat{G} = [X\rightarrow Y]$ (and $\hat{G} \neq [X\rightarrow Y]$ otherwise)
3. [Arbitrary units] 
$C(aX+b, cY+d) = C(X,Y); a,b,c,d\in R, a\neq 0, c\neq 0$

Learn Before

Related

Learn After