- Active days per user
- Revenue per user clicks
- Time active per vist

Example of OECs for online platforms

1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

Necessary ingredients for running useful controlled experiments

OECs should typically be normalized by the randomization unit. For example in comparing variants, the control group and the treatment groups to see which one generated more revenue, it would be a mistake to look at total revenue. The better OEC would be revenue-per-user. This is because even if the allocated traffic was equal, by chance (or on purpose) there can be differences in the sample sizes of the two (or more) groups.

Normalizing OECs

The preferred outcome measurement for an experiment. Different companies use different OECs based on their design objectives and business models. 

University of Michigan - Ann Arbor

- Overall evaluation criterion (OEC)
- Parameter
- Variant
- Randomization unit

List of Online Controlled Experiment Terminology

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

The variables that influence the outcome (overall evaluation criterion) or other other metrics of interest. 

Parameter (in online controlled experiments)

In online A/B testing (OCEs) the variants are the A and B, which are the control and treatment groups. They are the user experiences that are aiming to be measured/tested by finding the values of different parameters and outcome variables. 

Variant (in online controlled experiments)

The randomization process that is used to assign the different user experiences (variants) to different users (units). This is a very important process in controlling for confounders to isolate the causal effect.

Learn Before

Related

Learn After

Learn After

Related