Google

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Criticisms of Null Hypothesis Testing

A frequent misinterpretation of the $$p$$-value is the belief that it represents the probability that the null hypothesis is true. In reality, the $$p$$-value is defined as the probability of obtaining the observed sample result if the null hypothesis were true, rather than a measure of the truth of the hypothesis itself.

Misinterpreting p-values as the Probability of the Null Hypothesis

A common statistical misconception is the belief that $$1 - p$$ equals the probability that a statistically significant result will be successfully replicated. Researchers often over-optimistically assume that low $$p$$-values guarantee replication, while actual statistical power requirements for high replication probability are typically much more demanding.

Misinterpreting p-values as the Probability of Replication

The conventional practice of rejecting the null hypothesis when $$p < 0.05$$ is criticized for creating a mathematically arbitrary dividing line. This rigid threshold can lead to nearly identical findings—such as one study with $$p = 0.04$$ and another with $$p = 0.06$$—being treated as fundamentally different, with only the former deemed worthy of publication.

Arbitrary Nature of the 0.05 Significance Threshold

Critics argue that rejecting a null hypothesis is often uninformative because it only asserts that a relationship is 'nonzero' without describing its actual magnitude. Furthermore, since relationships in the population are rarely exactly zero, rejecting a null hypothesis may simply confirm a fact that was already likely to be true without providing new insight.

Limited Informativeness of Rejecting the Null Hypothesis

Researcher Robert Abelson defended null hypothesis testing as a 'principled argument' that serves an important purpose in science. He argued that it provides a systematic framework for researchers to demonstrate that their results are not mere chance occurrences, which is particularly vital when investigating and convincing others of new phenomena.

Abelson's Defense of Null Hypothesis Testing

In 2015, the journal 'Basic and Applied Social Psychology' implemented a formal ban on null hypothesis testing and $$p$$-values. The editors encouraged authors to focus on descriptive statistics and effect sizes instead, highlighting a significant challenge to the traditional statistical 'gold standard' in psychological research.

Learn Before

Related