- Active days per user
- Revenue per user clicks
- Time active per vist

University of Michigan - Ann Arbor

The preferred outcome measurement for an experiment. Different companies use different OECs based on their design objectives and business models. 

Overall evaluation criterion (OEC)

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

Example of OECs for online platforms

1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

Necessary ingredients for running useful controlled experiments

OECs should typically be normalized by the randomization unit. For example in comparing variants, the control group and the treatment groups to see which one generated more revenue, it would be a mistake to look at total revenue. The better OEC would be revenue-per-user. This is because even if the allocated traffic was equal, by chance (or on purpose) there can be differences in the sample sizes of the two (or more) groups.

Learn Before

Related