1. What is the randomization unit? (users, logins, communities, etc.)
2. What population (of the randomization unit) do we want to target?
3. How large does our experiment need to be? (sample size)
4. How long do we run the experiment? (time)

University of Michigan - Ann Arbor

Using RCTs in an online setting. Often deals with testing new features/hypothesis on digital platforms. Companies like microsoft, google and netflix regularly engage in them to improve revenue and UX.

Online Controlled Experiments

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

- Overall evaluation criterion (OEC)
- Parameter
- Variant
- Randomization unit

List of Online Controlled Experiment Terminology

1. The desire to make data-driven decisions and availability of a useful OEC

2. The willingness to invest in being able the infrastructure and tests that are needed 

3.  The organization assesses that they are poor at assessing the value of ideas 

Key tenets for organizations that wish to run online controlled experiments

Questions in Designing Online Experiments

https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf

Learn Before

Related