Regression based features represent the majority of features in algorithms using predefined features. They come in various forms, such as the errors of polynomial regressions of various degrees, independence of the residuals with the cause of the polynomial regression. 

Features of conditional distribution variability have been introduced by Fonollosa. One of those, called standard deviation of the conditional distributions (CDS) manages to achieve good performance even when used alone. The CDS score measures the spread of the conditional distributions after normalization of the bins:
$CDS(X,Y) = \sqrt{ \frac{1}{M} \sum_{y=0}^{M-1} var_x (p_n(Y=y|X=x))  }$
where $p_n ( Y = y | X = x )$ represents the normalized conditional probability and $var_x$ the sample variance over $x$ . 

This feature proved itself very useful for causality detection, as it captures the distribution asymmetry; typically, it standalone yields a score of 0.69 on the Tübingen dataset.

Regression Based Features

In the cause effect pairs challenge, many statistical quantities have been used to highlight patterns and asymmetries that might provide hints of the causal direction. These features come in different natures: information theory, regression based or statistical properties. The latter denotes features such as moments of the empirical distributions, and moments of regression residuals. These quantities are computed for the learning machine following in the pipeline to lever these features in order to detect causal patterns in the distributions.

University of Michigan - Ann Arbor

An intuitive way to obtain features out of empirical distributions for this new learning problem is to use the output of preexisting causal discovery algorithms, but also feature characterizing the joint and marginal distributions of the samples. There are different types of features that can be employed as features for the classifier, including:
 - Statistical Features of the Distribution
 - Statistical Asymmetries in the Distribution
  - Preexisting Pairwise Causal Discovery Algorithms
 - Applying Transformations to Variables

Handcrafted Causal Features

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Independence tests are always included in the algorithms, either to avoid testing for causal relationships if the variables are independent, or to maximize the accuracy of the predictions as a class in the challenge was dedicated to independent pairs. 

The independence test statistics used consist in mainly two types: the correlation-based and the kernel-based tests. Firstly, the correlation-based tests consist in the well-known statistic tests such as the Pearson’s correlation and the Spearman’s correlation, but also tests based on mutual information. The challenges contained all types of data, including continuous data. In order to compute mutual information out of the empirical distributions, algorithms binned their continuous variables prior to computing mutual information based features. Here we will consider U , V , obtained by binning X and Y . Examples of these features are 
 - mutual information
 - normalized mutual information 
 - adjusted mutual information.

Statistical Features of the Distribution

Statistical Asymmetries in the Distribution

Many approaches basing their inference on predefined features use as input of the classifier already known models for pairwise causal discovery in a stacked classifier fashion. 

Examples of used algorithms are the Additive Noise Model (ANM) model or the Information Geometric Causal Inference (IGCI) model. These two models are the most employed, as they represent a decent tradeoff between performance and computational cost. 

Preexisting Pairwise Causal Discovery Algorithms

Beyond computing all the features on the empirical distributions given as input, Almeida and Fonollosa also compute additional sets of the same features, but by changing the input variables by applying transformations. These transformations come in various forms, such as conversions from one type of variable to another, aggregating the distributions, or computing regressions and using the residuals as the new variables. 

This kind of transformations allow for computing higher order statistics and to grow the number of features considerably, as such transformations can be stacked multiple times before computing the various features.

Learn Before

Related

Learn After