Going from empirical distributions to a fixed-size vector representing the learnt relevant features implies a dimension reduction operation. Lopez-Paz et al. leverages mean embeddings to perform this operation: after applying a transformation to each point in the sample, all outputs are averaged to produce the feature vector representing the sample. This process can be summed up by the following equation:
$\mu(P_{S_j}) = \frac{1}{n_j} \sum_{i=1}^{n_j} f(x_i, y_i)$
where $f$ is a function with learnable parameters, $n_j$ is the number of points in the sample.

University of Michigan - Ann Arbor

Kernel embeddings allow for a general and strong representation of the distributions. However, these representation are not specific to the problem of pairwise causal discovery and therefore some patterns might be missed by those. 

Therefore, adapting the embeddings to the given distributions and to the task through learning allow the algorithms to automatically distinguish relevant patterns in the distributions.

 This paradigm fits with “Deep Learning” or more generally into the “Automatic Machine Learning” concept in which only the data has to be fed to the algorithm with no further domain specific knowledge. This merges representation learning and supervised learning: the algorithms learn their own features based on the given data and task.

Automatic Feature Construction out of Distributions

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

Learning an Custom Embedding from the Samples

Another idea to represent the empirical distribution into a fixed-size two dimensional object would be to represent the pair given as input as a scatter plot; the algorithms would then try to visually identify causal patterns in the drawn scatter plot. This approach, exploited by Singh et al. through a deep convolutional neural network, aligns itself with the examples and the idea that non-invertible causal mechanisms (a visually noticeable feature) give away the causal direction. 

Many different visual representations of the distributions are available to the practitioners, and little is known on their influence.Here are two of them: 
 - the “raw” scatter plot of the data, where a pixel is either 1 or 0 depending on whether a data point is present in the region represented by the pixel
 - a Gaussian distribution centered on each point, with a relatively low variance

Learn Before

Related