The preferred outcome measurement for an experiment. Different companies use different OECs based on their design objectives and business models. 

Overall evaluation criterion (OEC)

The variables that influence the outcome (overall evaluation criterion) or other other metrics of interest. 

Parameter (in online controlled experiments)

In online A/B testing (OCEs) the variants are the A and B, which are the control and treatment groups. They are the user experiences that are aiming to be measured/tested by finding the values of different parameters and outcome variables. 

Variant (in online controlled experiments)

The randomization process that is used to assign the different user experiences (variants) to different users (units). This is a very important process in controlling for confounders to isolate the causal effect.

Randomization unit (in online controlled experiments)

- Overall evaluation criterion (OEC)
- Parameter
- Variant
- Randomization unit

University of Michigan - Ann Arbor

Using RCTs in an online setting. Often deals with testing new features/hypothesis on digital platforms. Companies like microsoft, google and netflix regularly engage in them to improve revenue and UX.

Online Controlled Experiments

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

1. The desire to make data-driven decisions and availability of a useful OEC

2. The willingness to invest in being able the infrastructure and tests that are needed 

3.  The organization assesses that they are poor at assessing the value of ideas 

Key tenets for organizations that wish to run online controlled experiments

1. What is the randomization unit? (users, logins, communities, etc.)
2. What population (of the randomization unit) do we want to target?
3. How large does our experiment need to be? (sample size)
4. How long do we run the experiment? (time)

Questions in Designing Online Experiments

https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf

Learn Before

Related

Learn After

Learn After

Related