Cornfield's inequality was the first formal method for sensitivity analysis in observational studies. It provides a way to prevent causal relationships from being immediately refuted with the assertion that a new factor is the true cause. 

Cornfield's Inequality

Pearl, Judea, et al. “Sensitivity Analysis of Linear Structural Causal Models.”

http://proceedings.mlr.press/v97/cinelli19a/cinelli19a.pdf

Sensitivity Analysis of Linear Structural Causal Models

Informally, a model is identifiable if it is theoretically possible to learn the true values of this model's parameters after obtaining an infinite number of observations from it. 

Identification Problems are part of inferences, where we want to learn the true value of parameters from infinite observational data. When we cannot do it even through infinite data points, we fail to do identification.

Identification in Sensitivity Analysis

Sensitivity analysis refers to investigating whether the specified outcome could have resulted from alternative hypotheses and how strong those other hypotheses would need to be in order to explain the observed data.

The resulting relationships can be used to contrast the likelihoods with the original hypothesis. 

University of Michigan - Ann Arbor

A Randomized Controlled Trial (RCT) or a Controlled Experiment is a method to eliminate unmeasurable confounders effect on causal inference by implementing an intervention on a randomly selected group of individuals (treatment/intervened group) while keeping other individuals unchanged (controlled group, placebo, not-intervened group) to observe the changes in outcomes due to the intervention effect. The RCTs are intended to "disarm" confounders (confounding variables).

Randomized Controlled Trial (RCT) = Controlled Experiment

Confounders are variables that affect both the outcome and the factor we are trying to determine as the cause.  The confounders are what RCTs try to "disarm."

For example, when measuring the effect of Alcohol use (X) on Lung cancer (Y), Smoking (Z) may be a confounder because those who consume more alcohol also tend to smoke more. 

Confounders in Causal Inference

Pearl, J., & Mackenzie, D. (2018). $$\mathit{The \ Book \ of \ Why: \ The \ New \ Science \ of \ Cause \ and \ Effect.}$$ Basic Books.

The Book of Why

To better understand how we conduct RCT, here is a real example in the book explained in a simpler way.

For example we want to compare two Fertilizer. However, there are many confounders, including weather, soil and etc. Like the Causal Diagram below shows.

So in order to eliminate the confounders, you need to assign the fertilizers randomly into subplots on a land. In this way you can guarantee the majority time you assign fertilizers in a neither most fertile or least fertile subplots. 

Thus, the confounders can be eliminated. Right now, the uncertainty comes from randomlization process rather than the characteristics of the soil. 

Example of RCT

Sensitivity Analysis

A conventional measure of a treatment's effectiveness. It averages treatment efficacy over a population and can be found using a randomized control trial (RCT). It measures how much an increase in a single unit of treatment variable X will affect the outcome variable Y. When you have properly eliminated all confounding variables, you can obtain an estimate through running a regression.

Average Causal Effect (ACE)

Using RCTs in an online setting. Often deals with testing new features/hypothesis on digital platforms. Companies like microsoft, google and netflix regularly engage in them to improve revenue and UX.

Online Controlled Experiments

Efficacy is the performance of an intervention under ideal and controlled circumstances, whereas effectiveness is how well an intervention works in real-world environments. An intervention can be referred to as efficacious if it demonstrates significant results in controlled experiments, but cannot be referred to as effective until it demonstrates clinical or real-world use. 

Placebo-controlled RCTs are ideal for efficacy evaluation, whereas RCTs comparing an intervention with usual care practices is ideal for effectiveness evaluation.  

Effectiveness- vs. Efficacy-Oriented Randomized Control Trials

Often used in clinical trials, a double-blinded experiment is an experimental technique in which neither the investigators nor the subjects know who is receiving an intervention or not.  It is to prevent bias.  

Double-Blinded Experiment

1. There are experimental units (people, users, etc) that can be parts of different variants (treatment group or control group) that will not interfere (interact, communicate, etc) with one another 

2. There is a large enough sample size of experimental units (people, users, etc)

3. Key metrics (parameters, evaluation criterions, etc) can be calculated and represent the desired measurement

4. The treatment group is practical to make. For online experiments this could mean the software development is practical, and for medical experiments it could be whether it is reasonable to have humans try a potential drug

Necessary ingredients for running useful controlled experiments

A vegetarian boy, Daniel, tried to prove to the overseer of the King that being a vegetarian would not impair his ability to serve the king. For 10 days, the four of them including Daniel were given only vegetables, and the other group was given the king’s meat and wine. After 10 days, the two groups were compared. The result is that Daniel and his three companions grew up healthy on a vegetarian diet. 
- Two groups are comparable and can represent a population. 
- It is important to compare groups instead of individual comparisons.
- Two groups are selected in advance, instead of advertising, only the good ones are shown.

Example of Controlled experience

A case control study is an observational study method that selects people with a disease and those without a disease to investigate their exposure to a risk factor and its degree respectively, so as to determine whether the exposure risk factor is related to a disease and the degree of association.

Drawbacks: (Tobacco case as an example)
-  Retrospective: We know the subject has cancer, and in that context we have to go back and figure out why
-  Reversed logic of probability: What these data tell us is the probability that cancer patients are smokers, not the probability that smokers get cancer. For those who want to know if they should smoke, the probability of cancer in smokers is the probability they really care about
-  Recall bias: While the interviewers may be blinded to cancer vs. control, the patients themselves know whether or not they had cancer. Having already gone through this event can affect their recollections
-  Selection bias: Hospitalized cancer patients (aka those on record as having cancer and those specifically hospitalized for it) are not necessarily representative of the population at large or the smoking population at large.

Advantages of RCTs over Case Control Studies

Advantages of Case Control Studies:
- Quicker, cheaper and require less time and effort than cohort studies
- Case-control studies can study multiple risk factors/exposures
- They are useful for studying outcomes that take a long time to develop

Drawbacks of RCTs: 
- Ethics: RCT approaches are usually controversial and must be handled with care. The experimental group (intervened group) and control group (not- intervened group) are likely to have different outcomes and impact people who get involved in the trial. For example, you cannot assign people to smoke for decades just to figure out if they would get cancer. 
- Feasibility: Running RCTs can be very expensive. They cost time, effort and money. It can take decades to assess the long-term effect of a vaccine. Due to the high costs, most RCTS only inspect one or very few variables. Therefore they rarely reflect the full picture of a complicated issue.


Advantages of Case Control Studies over RCTs


In a randomized controlled trial (RCT), ensuring true randomness requires implementing procedures that minimize both allocation bias and selection bias. Alongside ensuring randomness, researchers must design the trial to maximize statistical power. While assigning participants to equal-sized groups generally maximizes statistical power, unequal group sizes can sometimes provide more statistical power for specific types of analyses.

Ensuring Randomness and Statistical Power in Randomized Controlled Trials

From the text:

A confounder of X and Y is a variable Z that:
1. is associated with X in the population at large
2. associated with Y among the people who have not been exposed to the treatment in X.
3. (Recently) Z should not be on the causal path between X and Y.

Classical Epidemiological Definition of a Confounder

When the junction $A \leftarrow B \rightarrow C$ appears in a Causal Diagram, we say that B is a confounder for both A and C. It might appear that there is a causal relationship between A and C, because there is an observable relationship between A and C when B is varied. However, if we condition on B ("hold B constant") we observe that A and C become conditionally independent. 

$A \leftarrow B \rightarrow C$: The Fork and Confouders

The following names have been used in different references to refer to "confounders": 
- "Lurking variables" (The Book of Why, p. 18) or "Suspected Lurking third variable" (The Book of Why, p. 138)
- "Common causes" (The Book of Why, p. 48)
- "Background factors" (The Book of Why, p. 48)
- "Mixing (of effects)" (The Book of Why, p. 138)
- "Good controls" (A Crash Course in Good and Bad
Controls, p. 1)

Other Names for "Confounders"

1- Bad controls
2- Good controls (also known as “confounders” or “deconfounders”)
3- Neutral controls

Types of control variables in causal inference

Causal Definition of Confounder

Prior to the advent of do-operators and causal diagrams, statisticians and researchers defined confounders based on surrogate definitions which were often unsatisfactory from a causal point of view. One surrogate definition was declarative, stating that "A confounder is any variable that is correlated with both X and Y." Another definition was procedural, stating that a confounder could be characterized by statistical testing. Either one leads to the existence of "surrogate confounders," which help to reduce, but not eliminate bias when determining relationships between variables. 

Previous Surrogate Definitions of Confounder 

VanderWeele, T. J., & Shpitser, I. (2013). On the definition of a confounder. Annals of statistics, 41(1), 196–220. https://doi.org/10.1214/12-aos1058
Link: https://bit.ly/3fTAeZY

On the definition of a confounder

Deconfounding is defined as mitigating the adverse effect of a confounder in causal inference.  There are multiple methods of deconfounding depending on whether the confounder is:
- Measurable
- Unmeasurable

Learn Before

Related

Learn After