In studies with p-values < 0.05, effect sizes are often inflated due to large observed differences. Therefore, studies with very low power cannot be generalized to the population of interest since the probability of committing type 1 error increases and is more likely to be greater than 5% when a p-value of less than 0.05 is used as a guideline for a hypothesis test. 

Effect Size in Studies Lacking Power

A numerical index, commonly represented as Pearson's $$r$$, that measures the direction and strength of a linear relationship between two quantitative variables. The coefficient is bounded between $$-1.00$$ and $$+1.00$$, where values further from zero indicate a tighter, more predictable association, and values near zero indicate a negligible relationship. In research, this coefficient frequently serves as an effect size measure to evaluate how strongly variables are connected in the population.

Correlation Coefficient

A widely adopted statistical metric used to quantify the effect size, or the standardized difference, between two independent group means. It is calculated by taking the absolute difference between the two sample means and dividing it by the standard deviation. Because it measures the actual magnitude of an experimental effect, researchers often report it alongside null hypothesis tests to provide a clearer picture of relationship strength.

Cohen's $$d$$

What does an effect size estimate in a statistical analysis?

A quantitative measure, such as Cohen's $$d$$ or Pearson's $$r$$, used to estimate the strength of a statistical relationship within a population. In response to criticisms of null hypothesis testing, researchers are encouraged to report effect sizes alongside their tests. This practice is crucial because a $$p$$-value alone only indicates whether a relationship is statistically significant—which is heavily influenced by sample size—but fails to convey the actual magnitude or importance of that relationship.

University of Michigan - Ann Arbor

Google

Components of Statistical Analysis

Null hypothesis testing is the most common approach to inferential statistics in psychological research. It provides researchers with a formal, structured process to evaluate whether an observed sample result likely occurred by random chance or whether it accurately reflects a genuine relationship or difference in the broader population.

Null Hypothesis Testing

Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6646942/

Power analysis and effect size in mixed effects models: A tutorial

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

The statistical power of a research design is the probability that the study will reject the null hypothesis, given a specific sample size and an expected relationship strength. For instance, if a design has a power of $$.80$$, it means there is an $$80\%$$ chance of correctly rejecting the null hypothesis when the expected relationship exists in the population. It is essentially the complement of the probability of making a Type II error, so researchers strive for high statistical power to avoid missing a real effect.

Statistical Power

Effect Size

The $$t$$-test is the most frequently used null hypothesis test for evaluating the statistical relationship between two means. There are three primary variations of this procedure—the one-sample $$t$$-test, the dependent-samples $$t$$-test, and the independent-samples $$t$$-test—each designed to accommodate slightly different research designs.

t-Test

In null hypothesis testing, researchers make one of two formal decisions (to reject or retain the null hypothesis) concerning one of two possible states of reality (the null hypothesis is actually true or false). This intersection creates a matrix of four distinct possible outcomes. Two outcomes are correct decisions: rejecting a false null hypothesis and retaining a true one. The remaining two outcomes are statistical errors: rejecting a true null hypothesis (Type I error) and retaining a false null hypothesis (Type II error).

Outcomes of a Null Hypothesis Test

A major criticism of null hypothesis testing is that it is fundamentally uninformative. Because the null hypothesis posits that there is precisely zero relationship between variables in the population, critics argue that it is never literally true in the real world. Consequently, rejecting the null hypothesis merely confirms that some nonzero relationship exists, which provides researchers with very little precise information about the actual strength or nature of the phenomenon being studied.

Informativeness of Null Hypothesis Testing

Despite widespread criticisms, some statisticians and researchers strongly defend null hypothesis testing when it is understood and applied correctly. A primary argument in its defense is that it provides a standardized, principled method for researchers to demonstrate that their findings are not merely the result of random chance. This framework is considered especially useful and necessary when investigating new or unestablished scientific phenomena.

Defense of Null Hypothesis Testing

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

A statistical range calculated around a sample statistic designed to contain the true population parameter a specified percentage of the time, most commonly $$95\%$$. Some researchers advocate using confidence intervals as an alternative to traditional null hypothesis testing because they are often more intuitive to interpret. Furthermore, a confidence interval inherently provides the data needed to perform a null hypothesis test; if a hypothetical population mean falls completely outside the calculated interval, the sample mean is considered statistically significantly different from that hypothetical value.

Confidence Interval

In null hypothesis testing, researchers reject the null hypothesis in favor of the alternative hypothesis if the observed sample relationship would be extremely unlikely to occur under the assumption that the null hypothesis is true. This decision indicates there is sufficient statistical evidence to conclude that a relationship exists in the population.

Reject the Null Hypothesis

The $$t$$-test is the most commonly used null hypothesis test for evaluating statistical relationships that focus on the difference between two means.

$$t$$-Test

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Criticisms of Null Hypothesis Testing

To illustrate the logic of null hypothesis testing, consider Mehl and colleagues' investigation into whether men and women differ in talkativeness. Assuming the null hypothesis (that there is no difference in the population) was true, they evaluated the likelihood of finding their sample's small difference of $$d = 0.06$$. Finding this result fairly likely under the null hypothesis, they decided to retain the null hypothesis, concluding there was no evidence of a sex difference in the population.

Mehl's Study on Talkativeness

Kanner and colleagues provided an example of rejecting the null hypothesis by studying the relationship between daily hassles and psychological symptoms. They asked how likely it would be to find their sample's strong correlation of $$+.60$$ if the null hypothesis (that there is no correlation in the population) were true. Because such a strong sample relationship would be fairly unlikely under the null hypothesis, they rejected the null hypothesis in favor of the alternative hypothesis, concluding a positive correlation exists in the population.

Kanner's Study on Hassles and Symptoms

Null hypothesis testing follows a structured logic: researchers first assume the null hypothesis is true, calculate how likely their sample results would be under this assumption, and then decide to either reject or retain the null hypothesis based on this calculated probability.

Learn Before

Related

Learn After