1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

Necessary ingredients for running useful controlled experiments

The randomization process that is used to assign the different user experiences (variants) to different users (units). This is a very important process in controlling for confounders to isolate the causal effect.

University of Michigan - Ann Arbor

- Overall evaluation criterion (OEC)
- Parameter
- Variant
- Randomization unit

List of Online Controlled Experiment Terminology

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

The preferred outcome measurement for an experiment. Different companies use different OECs based on their design objectives and business models. 

Overall evaluation criterion (OEC)

The variables that influence the outcome (overall evaluation criterion) or other other metrics of interest. 

Parameter (in online controlled experiments)

In online A/B testing (OCEs) the variants are the A and B, which are the control and treatment groups. They are the user experiences that are aiming to be measured/tested by finding the values of different parameters and outcome variables. 

Learn Before

Related

Learn After