 - KISS approach ( Keep It Simple, Stupid ) was proposed by Axelrod and Hamilton and stated that the models should include the minimal number of parameters and mechanisms and only add new ones only if required. 
 - On the other hand, the KIDS principle ( Keep It Descriptive, Stupid ) is an anti-simplistic approach that favors the accuracy of the model and states that all parameters and mechanisms that appear relevant should be included, but parameters that do not add quality of the model should be removed.

KISS & KIDS Approaches

When transposing these modeling approaches to the cause-effect problem, we can see that some methods favor models with low complexity and low explanatory power such as linear Gaussian models while others have a better explanatory power as they can model complex interactions between noise and causes (for example when using generative neural networks), but at the cost of more complex mechanisms. 

In the cause-effect pair literature, we can distinguish three different approaches used to deal with this complexity/fit trade-off:
 - Method 1: Choose best fit.
 - Method 2: Choose best Complexity
 - Method 3: Weighted bi-crateria aggregation between fit and complexity

Complexity/Fit Trade-off in Cause-Effect Pair Problem

Since we have defined the complexity $C_{\hat{\mathcal{B}}}(\theta)$ and the fit $S_{\hat{\mathcal{B}}}(\theta)$ of a candidate bivariate model parameterized by the vector of parameters θ . From a general point of view, we can now frame the model selection as a bi-objective trade-off with a Pareto front of optimal models.

This general bi-objective trade-off between fit and complexity is very general in science. It has been seen from two different angles. Some of the modeling approaches favor models of lower complexity, while others favor models with better explanatory power:
 - KISS Approach (Keep It Simple, Stupid)
 - KIDS Approach (Keep It Descriptive, Stupid)

University of Michigan - Ann Arbor

The determination of the best explainable model is based in the literature on these main lines: 
 - Different candidate bivariate models (hypotheses) are evaluated in both directions.
 - For each candidate model one evaluates a score monitoring the trade-off between the fit of the model (meaning its adequacy to the observational data) and the complexity of the mechanisms involved. 
 - The model with the best score is returned, with its corresponding causal arrow X → Y or Y → X .

Determination of the Best Explainable Model

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

In order to model such continuous underlying bivariate generative process $\mathcal{B}_{\mathcal{G}, f, P_N}$ which is assumed to have generated the data, the author introduce the notion of candidate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}$ described by: 
 - a structure defined by a causal orientation X → Y or Y → X 
 - its estimated mechanisms modeled by $\hat{f}$
 -  an estimated distribution of noise $Q_N$ 

Candidate Bivariate Models

In order to evaluate the quality of a candidate generative model, we introduce a fit score $S_{\hat{\mathcal{B}}} (\theta)$ of a candidate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}(\theta)$. This score has been implemented in different ways in the literature. It has always the property to be minimal when P = Q (perfect fit in the large sample limit).

Here are some example of scores:
 - Log-Likelihood Parametric Scores
 - Implicit Fit Score Computed as an Independence Score Between Cause and Noise Variable
 - Non-parametric Scores

Model Fitness Score

The authors introduce a term of complexity of a candidate bivariate model $\hat{\mathcal{B}}_{\hat{\mathcal{G}}, \hat{f}, Q_N}(\theta)$ that we note $C_{\hat{B}}(\theta)$. This complexity notion refers to a simplicity prior on the underlying data generative process. It has been handled in different ways in the literature from fix a class of admissible mechanisms to more flexible criteria.

Th ecomplexity term can be defined through different approaches, including:
 - Explicit Class of Admissible Mechanisms
 - Kolmogorov Complexity

Learn Before

Related

Learn After