A frequent misinterpretation of the $$p$$-value is the belief that it represents the probability that the null hypothesis is true. In reality, the $$p$$-value is defined as the probability of obtaining the observed sample result if the null hypothesis were true, rather than a measure of the truth of the hypothesis itself.

Misinterpreting p-values as the Probability of the Null Hypothesis

A common statistical misconception is the belief that $$1 - p$$ equals the probability that a statistically significant result will be successfully replicated. Researchers often over-optimistically assume that low $$p$$-values guarantee replication, while actual statistical power requirements for high replication probability are typically much more demanding.

Misinterpreting p-values as the Probability of Replication

The conventional practice of rejecting the null hypothesis when $$p < 0.05$$ is criticized for creating a mathematically arbitrary dividing line. This rigid threshold can lead to nearly identical findings—such as one study with $$p = 0.04$$ and another with $$p = 0.06$$—being treated as fundamentally different, with only the former deemed worthy of publication.

Arbitrary Nature of the 0.05 Significance Threshold

Critics argue that rejecting a null hypothesis is often uninformative because it only asserts that a relationship is 'nonzero' without describing its actual magnitude. Furthermore, since relationships in the population are rarely exactly zero, rejecting a null hypothesis may simply confirm a fact that was already likely to be true without providing new insight.

Limited Informativeness of Rejecting the Null Hypothesis

Researcher Robert Abelson defended null hypothesis testing as a 'principled argument' that serves an important purpose in science. He argued that it provides a systematic framework for researchers to demonstrate that their results are not mere chance occurrences, which is particularly vital when investigating and convincing others of new phenomena.

Abelson's Defense of Null Hypothesis Testing

In 2015, the journal 'Basic and Applied Social Psychology' implemented a formal ban on null hypothesis testing and $$p$$-values. The editors encouraged authors to focus on descriptive statistics and effect sizes instead, highlighting a significant challenge to the traditional statistical 'gold standard' in psychological research.

The 2015 Ban on p-values in Basic and Applied Social Psychology

Which of the following is considered a major criticism of null hypothesis testing?

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Google

In psychological research, a finding is typically considered 'statistically significant' if there is a probability of five percent or less that the observed difference between groups occurred by random chance. This standard helps researchers confidently conclude that the results are meaningful and not just a fluke.

Statistical Significance in Psychology

Null hypothesis testing is the most common approach to inferential statistics in psychological research. It provides researchers with a formal, structured process to evaluate whether an observed sample result likely occurred by random chance or whether it accurately reflects a genuine relationship or difference in the broader population.

Null Hypothesis Testing

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

A primary goal in statistical analysis is to limit the rate of 'false positive' claims, which are erroneous conclusions that an effect exists when it does not. Psychologists achieve this by setting a strict criterion for statistical significance, commonly accepting a 5% or lower probability that the observed results are due to chance.

Limiting False Positives in Research

If an experiment evaluates a nutritional supplement and finds that participants taking a placebo achieve the same outcomes as those taking the active supplement, the difference is not meaningful. In this scenario, statistical analysis would show the result is likely due to chance, leading to the conclusion that the supplement is ineffective.

Example of a Non-Significant Result: Nutritional Supplement

Criticisms of Null Hypothesis Testing

In psychological research, a finding is typically considered 'statistically significant' if the probability that the observed difference occurred by random chance is:

The $$t$$-test is the most frequently used null hypothesis test for evaluating the statistical relationship between two means. There are three primary variations of this procedure—the one-sample $$t$$-test, the dependent-samples $$t$$-test, and the independent-samples $$t$$-test—each designed to accommodate slightly different research designs.

t-Test

In null hypothesis testing, researchers make one of two formal decisions (to reject or retain the null hypothesis) concerning one of two possible states of reality (the null hypothesis is actually true or false). This intersection creates a matrix of four distinct possible outcomes. Two outcomes are correct decisions: rejecting a false null hypothesis and retaining a true one. The remaining two outcomes are statistical errors: rejecting a true null hypothesis (Type I error) and retaining a false null hypothesis (Type II error).

Outcomes of a Null Hypothesis Test

A major criticism of null hypothesis testing is that it is fundamentally uninformative. Because the null hypothesis posits that there is precisely zero relationship between variables in the population, critics argue that it is never literally true in the real world. Consequently, rejecting the null hypothesis merely confirms that some nonzero relationship exists, which provides researchers with very little precise information about the actual strength or nature of the phenomenon being studied.

Informativeness of Null Hypothesis Testing

Despite widespread criticisms, some statisticians and researchers strongly defend null hypothesis testing when it is understood and applied correctly. A primary argument in its defense is that it provides a standardized, principled method for researchers to demonstrate that their findings are not merely the result of random chance. This framework is considered especially useful and necessary when investigating new or unestablished scientific phenomena.

Defense of Null Hypothesis Testing

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

A quantitative measure, such as Cohen's $$d$$ or Pearson's $$r$$, used to estimate the strength of a statistical relationship within a population. In response to criticisms of null hypothesis testing, researchers are encouraged to report effect sizes alongside their tests. This practice is crucial because a $$p$$-value alone only indicates whether a relationship is statistically significant—which is heavily influenced by sample size—but fails to convey the actual magnitude or importance of that relationship.

Effect Size

A statistical range calculated around a sample statistic designed to contain the true population parameter a specified percentage of the time, most commonly $$95\%$$. Some researchers advocate using confidence intervals as an alternative to traditional null hypothesis testing because they are often more intuitive to interpret. Furthermore, a confidence interval inherently provides the data needed to perform a null hypothesis test; if a hypothetical population mean falls completely outside the calculated interval, the sample mean is considered statistically significantly different from that hypothetical value.

Confidence Interval

In null hypothesis testing, researchers reject the null hypothesis in favor of the alternative hypothesis if the observed sample relationship would be extremely unlikely to occur under the assumption that the null hypothesis is true. This decision indicates there is sufficient statistical evidence to conclude that a relationship exists in the population.

Reject the Null Hypothesis

The $$t$$-test is the most commonly used null hypothesis test for evaluating statistical relationships that focus on the difference between two means.

$$t$$-Test

To illustrate the logic of null hypothesis testing, consider Mehl and colleagues' investigation into whether men and women differ in talkativeness. Assuming the null hypothesis (that there is no difference in the population) was true, they evaluated the likelihood of finding their sample's small difference of $$d = 0.06$$. Finding this result fairly likely under the null hypothesis, they decided to retain the null hypothesis, concluding there was no evidence of a sex difference in the population.

Mehl's Study on Talkativeness

Kanner and colleagues provided an example of rejecting the null hypothesis by studying the relationship between daily hassles and psychological symptoms. They asked how likely it would be to find their sample's strong correlation of $$+.60$$ if the null hypothesis (that there is no correlation in the population) were true. Because such a strong sample relationship would be fairly unlikely under the null hypothesis, they rejected the null hypothesis in favor of the alternative hypothesis, concluding a positive correlation exists in the population.

Kanner's Study on Hassles and Symptoms

Null hypothesis testing follows a structured logic: researchers first assume the null hypothesis is true, calculate how likely their sample results would be under this assumption, and then decide to either reject or retain the null hypothesis based on this calculated probability.

Learn Before

Related

Learn After