- Overall evaluation criterion (OEC)
- Parameter
- Variant
- Randomization unit

List of Online Controlled Experiment Terminology

1. The desire to make data-driven decisions and availability of a useful OEC

2. The willingness to invest in being able the infrastructure and tests that are needed 

3.  The organization assesses that they are poor at assessing the value of ideas 

Key tenets for organizations that wish to run online controlled experiments

1. What is the randomization unit? (users, logins, communities, etc.)
2. What population (of the randomization unit) do we want to target?
3. How large does our experiment need to be? (sample size)
4. How long do we run the experiment? (time)

Questions in Designing Online Experiments

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf

Practical Guide to Controlled Experiments on the Web


Using RCTs in an online setting. Often deals with testing new features/hypothesis on digital platforms. Companies like microsoft, google and netflix regularly engage in them to improve revenue and UX.

University of Michigan - Ann Arbor

A Randomized Controlled Trial (RCT) or a Controlled Experiment is a method to eliminate unmeasurable confounders effect on causal inference by implementing an intervention on a randomly selected group of individuals (treatment/intervened group) while keeping other individuals unchanged (controlled group, placebo, not-intervened group) to observe the changes in outcomes due to the intervention effect. The RCTs are intended to "disarm" confounders (confounding variables).

Randomized Controlled Trial (RCT) = Controlled Experiment

To better understand how we conduct RCT, here is a real example in the book explained in a simpler way.

For example we want to compare two Fertilizer. However, there are many confounders, including weather, soil and etc. Like the Causal Diagram below shows.

So in order to eliminate the confounders, you need to assign the fertilizers randomly into subplots on a land. In this way you can guarantee the majority time you assign fertilizers in a neither most fertile or least fertile subplots. 

Thus, the confounders can be eliminated. Right now, the uncertainty comes from randomlization process rather than the characteristics of the soil. 

Example of RCT

Sensitivity analysis refers to investigating whether the specified outcome could have resulted from alternative hypotheses and how strong those other hypotheses would need to be in order to explain the observed data.

The resulting relationships can be used to contrast the likelihoods with the original hypothesis. 

Sensitivity Analysis

A conventional measure of a treatment's effectiveness. It averages treatment efficacy over a population and can be found using a randomized control trial (RCT). It measures how much an increase in a single unit of treatment variable X will affect the outcome variable Y. When you have properly eliminated all confounding variables, you can obtain an estimate through running a regression.

Average Causal Effect (ACE)

Online Controlled Experiments

Efficacy is the performance of an intervention under ideal and controlled circumstances, whereas effectiveness is how well an intervention works in real-world environments. An intervention can be referred to as efficacious if it demonstrates significant results in controlled experiments, but cannot be referred to as effective until it demonstrates clinical or real-world use. 

Placebo-controlled RCTs are ideal for efficacy evaluation, whereas RCTs comparing an intervention with usual care practices is ideal for effectiveness evaluation.  

Effectiveness- vs. Efficacy-Oriented Randomized Control Trials

Often used in clinical trials, a double-blinded experiment is an experimental technique in which neither the investigators nor the subjects know who is receiving an intervention or not.  It is to prevent bias.  

Double-Blinded Experiment

1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

Necessary ingredients for running useful controlled experiments

A vegetarian boy, Daniel, tried to prove to the overseer of the King that being a vegetarian would not impair his ability to serve the king. For 10 days, the four of them including Daniel were given only vegetables, and the other group was given the king’s meat and wine. After 10 days, the two groups were compared. The result is that Daniel and his three companions grew up healthy on a vegetarian diet. 
- Two groups are comparable and can represent a population. 
- It is important to compare groups instead of individual comparisons.
- Two groups are selected in advance, instead of advertising, only the good ones are shown.

Example of Controlled experience

A case control study is an observational study method that selects people with a disease and those without a disease to investigate their exposure to a risk factor and its degree respectively, so as to determine whether the exposure risk factor is related to a disease and the degree of association.

Drawbacks: (Tobacco case as an example)
-  Retrospective: We know the subject has cancer, and in that context we have to go back and figure out why
-  Reversed logic of probability: What these data tell us is the probability that cancer patients are smokers, not the probability that smokers get cancer. For those who want to know if they should smoke, the probability of cancer in smokers is the probability they really care about
-  Recall bias: While the interviewers may be blinded to cancer vs. control, the patients themselves know whether or not they had cancer. Having already gone through this event can affect their recollections
-  Selection bias: Hospitalized cancer patients (aka those on record as having cancer and those specifically hospitalized for it) are not necessarily representative of the population at large or the smoking population at large.

Advantages of RCTs over Case Control Studies

Advantages of Case Control Studies:
- Quicker, cheaper and require less time and effort than cohort studies
- Case-control studies can study multiple risk factors/exposures
- They are useful for studying outcomes that take a long time to develop

Drawbacks of RCTs: 
- Ethics: RCT approaches are usually controversial and must be handled with care. The experimental group (intervened group) and control group (not- intervened group) are likely to have different outcomes and impact people who get involved in the trial. For example, you cannot assign people to smoke for decades just to figure out if they would get cancer. 
- Feasibility: Running RCTs can be very expensive. They cost time, effort and money. It can take decades to assess the long-term effect of a vaccine. Due to the high costs, most RCTS only inspect one or very few variables. Therefore they rarely reflect the full picture of a complicated issue.


Advantages of Case Control Studies over RCTs


In a randomized controlled trial (RCT), ensuring true randomness requires implementing procedures that minimize both allocation bias and selection bias. Alongside ensuring randomness, researchers must design the trial to maximize statistical power. While assigning participants to equal-sized groups generally maximizes statistical power, unequal group sizes can sometimes provide more statistical power for specific types of analyses.

Learn Before

Related

Learn After