1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

University of Michigan - Ann Arbor

The randomization process that is used to assign the different user experiences (variants) to different users (units). This is a very important process in controlling for confounders to isolate the causal effect.

Randomization unit (in online controlled experiments)

In online A/B testing (OCEs) the variants are the A and B, which are the control and treatment groups. They are the user experiences that are aiming to be measured/tested by finding the values of different parameters and outcome variables. 

Variant (in online controlled experiments)

The variables that influence the outcome (overall evaluation criterion) or other other metrics of interest. 

Parameter (in online controlled experiments)

The preferred outcome measurement for an experiment. Different companies use different OECs based on their design objectives and business models. 

Overall evaluation criterion (OEC)

A Randomized Controlled Trial (RCT) or a Controlled Experiment is a method to eliminate unmeasurable confounders effect on causal inference by implementing an intervention on a randomly selected group of individuals (treatment/intervened group) while keeping other individuals unchanged (controlled group, placebo, not-intervened group) to observe the changes in outcomes due to the intervention effect. The RCTs are intended to "disarm" confounders (confounding variables).

Randomized Controlled Trial (RCT) = Controlled Experiment

Kohavi, Ron, et al. Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing. Cambridge University Press, 2020.

Trustworthy Online Experiments

Necessary ingredients for running useful controlled experiments

- Active days per user
- Revenue per user clicks
- Time active per vist

Example of OECs for online platforms

OECs should typically be normalized by the randomization unit. For example in comparing variants, the control group and the treatment groups to see which one generated more revenue, it would be a mistake to look at total revenue. The better OEC would be revenue-per-user. This is because even if the allocated traffic was equal, by chance (or on purpose) there can be differences in the sample sizes of the two (or more) groups.

Normalizing OECs

To ensure randomness we need to make the following conditions
- minimize allocation bias
- minimize selection bias
- Maximize statistical power :
    - Generally, equal group sizes maximize statistical power, however, unequal groups sizes may be more powerful for some analyses

How do we ensure randomness in a RCT ?

To better understand how we conduct RCT, here is a real example in the book explained in a simpler way.

For example we want to compare two Fertilizer. However, there are many confounders, including weather, soil and etc. Like the Causal Diagram below shows.

So in order to eliminate the confounders, you need to assign the fertilizers randomly into subplots on a land. In this way you can guarantee the majority time you assign fertilizers in a neither most fertile or least fertile subplots. 

Thus, the confounders can be eliminated. Right now, the uncertainty comes from randomlization process rather than the characteristics of the soil. 

Example of RCT

Sensitivity analysis refers to investigating whether the specified outcome could have resulted from alternative hypotheses and how strong those other hypotheses would need to be in order to explain the observed data.

The resulting relationships can be used to contrast the likelihoods with the original hypothesis. 

Sensitivity Analysis

A conventional measure of a treatment's effectiveness. It averages treatment efficacy over a population and can be found using a randomized control trial (RCT). It measures how much an increase in a single unit of treatment variable X will affect the outcome variable Y. When you have properly eliminated all confounding variables, you can obtain an estimate through running a regression.

Average Causal Effect (ACE)

Using RCTs in an online setting. Often deals with testing new features/hypothesis on digital platforms. Companies like microsoft, google and netflix regularly engage in them to improve revenue and UX.

Online Controlled Experiments

Efficacy is the performance of an intervention under ideal and controlled circumstances, whereas effectiveness is how well an intervention works in real-world environments. An intervention can be referred to as efficacious if it demonstrates significant results in controlled experiments, but cannot be referred to as effective until it demonstrates clinical or real-world use. 

Placebo-controlled RCTs are ideal for efficacy evaluation, whereas RCTs comparing an intervention with usual care practices is ideal for effectiveness evaluation.  

Effectiveness- vs. Efficacy-Oriented Randomized Control Trials

Often used in clinical trials, a double-blinded experiment is an experimental technique in which neither the investigators nor the subjects know who is receiving an intervention or not.  It is to prevent bias.  

Double-Blinded Experiment

A vegetarian boy, Daniel, tried to prove to the overseer of the King that being a vegetarian would not impair his ability to serve the king. For 10 days, the four of them including Daniel were given only vegetables, and the other group was given the king’s meat and wine. After 10 days, the two groups were compared. The result is that Daniel and his three companions grew up healthy on a vegetarian diet. 
- Two groups are comparable and can represent a population. 
- It is important to compare groups instead of individual comparisons.
- Two groups are selected in advance, instead of advertising, only the good ones are shown.

Example of Controlled experience

A case control study is an observational study method that selects people with a disease and those without a disease to investigate their exposure to a risk factor and its degree respectively, so as to determine whether the exposure risk factor is related to a disease and the degree of association.

Drawbacks: (Tobacco case as an example)
-  Retrospective: We know the subject has cancer, and in that context we have to go back and figure out why
-  Reversed logic of probability: What these data tell us is the probability that cancer patients are smokers, not the probability that smokers get cancer. For those who want to know if they should smoke, the probability of cancer in smokers is the probability they really care about
-  Recall bias: While the interviewers may be blinded to cancer vs. control, the patients themselves know whether or not they had cancer. Having already gone through this event can affect their recollections
-  Selection bias: Hospitalized cancer patients (aka those on record as having cancer and those specifically hospitalized for it) are not necessarily representative of the population at large or the smoking population at large.

Advantages of RCTs over Case Control Studies

Advantages of Case Control Studies:
- Quicker, cheaper and require less time and effort than cohort studies
- Case-control studies can study multiple risk factors/exposures
- They are useful for studying outcomes that take a long time to develop

Drawbacks of RCTs: 
- Ethics: RCT approaches are usually controversial and must be handled with care. The experimental group (intervened group) and control group (not- intervened group) are likely to have different outcomes and impact people who get involved in the trial. For example, you cannot assign people to smoke for decades just to figure out if they would get cancer. 
- Feasibility: Running RCTs can be very expensive. They cost time, effort and money. It can take decades to assess the long-term effect of a vaccine. Due to the high costs, most RCTS only inspect one or very few variables. Therefore they rarely reflect the full picture of a complicated issue.


Learn Before

Related