Independence tests are always included in the algorithms, either to avoid testing for causal relationships if the variables are independent, or to maximize the accuracy of the predictions as a class in the challenge was dedicated to independent pairs. 

The independence test statistics used consist in mainly two types: the correlation-based and the kernel-based tests. Firstly, the correlation-based tests consist in the well-known statistic tests such as the Pearson’s correlation and the Spearman’s correlation, but also tests based on mutual information. The challenges contained all types of data, including continuous data. In order to compute mutual information out of the empirical distributions, algorithms binned their continuous variables prior to computing mutual information based features. Here we will consider U , V , obtained by binning X and Y . Examples of these features are 
 - mutual information
 - normalized mutual information 
 - adjusted mutual information.

Statistical Features of the Distribution

In the cause effect pairs challenge, many statistical quantities have been used to highlight patterns and asymmetries that might provide hints of the causal direction. These features come in different natures: information theory, regression based or statistical properties. The latter denotes features such as moments of the empirical distributions, and moments of regression residuals. These quantities are computed for the learning machine following in the pipeline to lever these features in order to detect causal patterns in the distributions.

Statistical Asymmetries in the Distribution

Many approaches basing their inference on predefined features use as input of the classifier already known models for pairwise causal discovery in a stacked classifier fashion. 

Examples of used algorithms are the Additive Noise Model (ANM) model or the Information Geometric Causal Inference (IGCI) model. These two models are the most employed, as they represent a decent tradeoff between performance and computational cost. 

Preexisting Pairwise Causal Discovery Algorithms

Beyond computing all the features on the empirical distributions given as input, Almeida and Fonollosa also compute additional sets of the same features, but by changing the input variables by applying transformations. These transformations come in various forms, such as conversions from one type of variable to another, aggregating the distributions, or computing regressions and using the residuals as the new variables. 

This kind of transformations allow for computing higher order statistics and to grow the number of features considerably, as such transformations can be stacked multiple times before computing the various features.

Applying Transformations to Variables

An intuitive way to obtain features out of empirical distributions for this new learning problem is to use the output of preexisting causal discovery algorithms, but also feature characterizing the joint and marginal distributions of the samples. There are different types of features that can be employed as features for the classifier, including:
 - Statistical Features of the Distribution
 - Statistical Asymmetries in the Distribution
  - Preexisting Pairwise Causal Discovery Algorithms
 - Applying Transformations to Variables

University of Michigan - Ann Arbor

In order to apply regular learning tools for classification, features have to be extracted out of the data distribution samples. This step is a feature construction step, and the literature has taken three different approaches to extract such features: 
 - Manual construction of causally relevant features to classify the pairs
 - Use embedding of the sample distributions into a fixed size feature vector: the resulting manifold will be mapped to the target classes using the training set, allowing to classify unseen examples using the same embedding. 
 - Not only use embeddings of distributions, but to also automatically learn and identify classification patterns using the training set.

Feature Construction out of Distributions for Pairwise Classification

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Handcrafted Causal Features

Another approach to feature construction for pairwise causal discovery is to use distribution embeddings to represent the distribution samples in a latent space as a vector with a fixed number of features. 

Unlike computing a custom set of variables, this approach represents each distribution in a latent space and the learning algorithm learns to split this latent space into the different classes. Inference of unseen pairs consist in applying the embedding to the distribution and reporting the label assigned to the region in the latent space corresponding to the image of the sample. One could see this operation as to look for the closest distribution in the training set to the sample and assign its label. 

Here are two types of embeddings: 
 - kernel-based embeddings of the joint distribution 
 - embeddings of the conditional distributions

Building Distribution Embeddings

Kernel embeddings allow for a general and strong representation of the distributions. However, these representation are not specific to the problem of pairwise causal discovery and therefore some patterns might be missed by those. 

Therefore, adapting the embeddings to the given distributions and to the task through learning allow the algorithms to automatically distinguish relevant patterns in the distributions.

 This paradigm fits with “Deep Learning” or more generally into the “Automatic Machine Learning” concept in which only the data has to be fed to the algorithm with no further domain specific knowledge. This merges representation learning and supervised learning: the algorithms learn their own features based on the given data and task.

Learn Before

Related

Learn After