 - KISS approach ( Keep It Simple, Stupid ) was proposed by Axelrod and Hamilton and stated that the models should include the minimal number of parameters and mechanisms and only add new ones only if required. 
 - On the other hand, the KIDS principle ( Keep It Descriptive, Stupid ) is an anti-simplistic approach that favors the accuracy of the model and states that all parameters and mechanisms that appear relevant should be included, but parameters that do not add quality of the model should be removed.

University of Michigan - Ann Arbor

Since we have defined the complexity $C_{\hat{\mathcal{B}}}(\theta)$ and the fit $S_{\hat{\mathcal{B}}}(\theta)$ of a candidate bivariate model parameterized by the vector of parameters θ . From a general point of view, we can now frame the model selection as a bi-objective trade-off with a Pareto front of optimal models.

This general bi-objective trade-off between fit and complexity is very general in science. It has been seen from two different angles. Some of the modeling approaches favor models of lower complexity, while others favor models with better explanatory power:
 - KISS Approach (Keep It Simple, Stupid)
 - KIDS Approach (Keep It Descriptive, Stupid)

Bi-objective Trade-Off between Fit and Complexity

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

KISS & KIDS Approaches

When transposing these modeling approaches to the cause-effect problem, we can see that some methods favor models with low complexity and low explanatory power such as linear Gaussian models while others have a better explanatory power as they can model complex interactions between noise and causes (for example when using generative neural networks), but at the cost of more complex mechanisms. 

In the cause-effect pair literature, we can distinguish three different approaches used to deal with this complexity/fit trade-off:
 - Method 1: Choose best fit.
 - Method 2: Choose best Complexity
 - Method 3: Weighted bi-crateria aggregation between fit and complexity

Learn Before

Related