Benchmark organizers aim at organizing a large scale evaluation of cause-effect pair methods, so they make efforts on creating good benchmark datasets.

The options to get benchmark data include:
 - Facilitate the collection of real data
 - Create synthetic data
 - Generate artificial data

University of Michigan - Ann Arbor

While talking about methods of evaluation of cause-effect pair algorithms, there are three points of view : 
 - that of algorithm developers in need to characterize their new algorithms both theoretically and empirically
 - that of practitioners who want to ensure the validity of a potential discovery
 - that of benchmark organizers who want to make fair comparisons.

Evaluation Methods for Cause-Effect Pair Algorithms

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

When it comes to the methods of evaluation of cause-effect pair algorithms,  the algorithm developers want to characterize their new algorithms both theoretically and empirically, so they care about:
 - backing algorithms with theoretical justifications (identifiability)
 - empirical assessment methods

Cause-Effect Pair Algorithm Evaluation Methods (for Algorithm Developers)

For practitioners who want to use causation coefficients in various real-world applications (in medicine or epidemiology, social sciences, econometrics, etc.), “evaluation”means assessing the reliability of given putative cause-effect relationship “discoveries” inferred from observational data. They do not want to evaluate methods, they want to use them to identify potential cause-effect relationships in pairs of real variables.

If practitioners are provided with a causation coefficient, they have expectation on:
 - Prerequisites
 - Efficacy
 - Post-hoc Evaluation

Learn Before

Related