The $$t$$-test is the most frequently used null hypothesis test for evaluating the statistical relationship between two means. There are three primary variations of this procedure—the one-sample $$t$$-test, the dependent-samples $$t$$-test, and the independent-samples $$t$$-test—each designed to accommodate slightly different research designs.

t-Test

In null hypothesis testing, researchers make one of two formal decisions (to reject or retain the null hypothesis) concerning one of two possible states of reality (the null hypothesis is actually true or false). This intersection creates a matrix of four distinct possible outcomes. Two outcomes are correct decisions: rejecting a false null hypothesis and retaining a true one. The remaining two outcomes are statistical errors: rejecting a true null hypothesis (Type I error) and retaining a false null hypothesis (Type II error).

Outcomes of a Null Hypothesis Test

A major criticism of null hypothesis testing is that it is fundamentally uninformative. Because the null hypothesis posits that there is precisely zero relationship between variables in the population, critics argue that it is never literally true in the real world. Consequently, rejecting the null hypothesis merely confirms that some nonzero relationship exists, which provides researchers with very little precise information about the actual strength or nature of the phenomenon being studied.

Informativeness of Null Hypothesis Testing

Despite widespread criticisms, some statisticians and researchers strongly defend null hypothesis testing when it is understood and applied correctly. A primary argument in its defense is that it provides a standardized, principled method for researchers to demonstrate that their findings are not merely the result of random chance. This framework is considered especially useful and necessary when investigating new or unestablished scientific phenomena.

Defense of Null Hypothesis Testing

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

A quantitative measure, such as Cohen's $$d$$ or Pearson's $$r$$, used to estimate the strength of a statistical relationship within a population. In response to criticisms of null hypothesis testing, researchers are encouraged to report effect sizes alongside their tests. This practice is crucial because a $$p$$-value alone only indicates whether a relationship is statistically significant—which is heavily influenced by sample size—but fails to convey the actual magnitude or importance of that relationship.

Effect Size

A statistical range calculated around a sample statistic designed to contain the true population parameter a specified percentage of the time, most commonly $$95\%$$. Some researchers advocate using confidence intervals as an alternative to traditional null hypothesis testing because they are often more intuitive to interpret. Furthermore, a confidence interval inherently provides the data needed to perform a null hypothesis test; if a hypothetical population mean falls completely outside the calculated interval, the sample mean is considered statistically significantly different from that hypothetical value.

Confidence Interval

In null hypothesis testing, researchers reject the null hypothesis in favor of the alternative hypothesis if the observed sample relationship would be extremely unlikely to occur under the assumption that the null hypothesis is true. This decision indicates there is sufficient statistical evidence to conclude that a relationship exists in the population.

Reject the Null Hypothesis

The $$t$$-test is the most commonly used null hypothesis test for evaluating statistical relationships that focus on the difference between two means.

$$t$$-Test

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Criticisms of Null Hypothesis Testing

To illustrate the logic of null hypothesis testing, consider Mehl and colleagues' investigation into whether men and women differ in talkativeness. Assuming the null hypothesis (that there is no difference in the population) was true, they evaluated the likelihood of finding their sample's small difference of $$d = 0.06$$. Finding this result fairly likely under the null hypothesis, they decided to retain the null hypothesis, concluding there was no evidence of a sex difference in the population.

Mehl's Study on Talkativeness

Kanner and colleagues provided an example of rejecting the null hypothesis by studying the relationship between daily hassles and psychological symptoms. They asked how likely it would be to find their sample's strong correlation of $$+.60$$ if the null hypothesis (that there is no correlation in the population) were true. Because such a strong sample relationship would be fairly unlikely under the null hypothesis, they rejected the null hypothesis in favor of the alternative hypothesis, concluding a positive correlation exists in the population.

Kanner's Study on Hassles and Symptoms

Null hypothesis testing follows a structured logic: researchers first assume the null hypothesis is true, calculate how likely their sample results would be under this assumption, and then decide to either reject or retain the null hypothesis based on this calculated probability.

Logic of Null Hypothesis Testing

What is the primary purpose of null hypothesis testing in psychological research?

Null hypothesis testing is the most common approach to inferential statistics in psychological research. It provides researchers with a formal, structured process to evaluate whether an observed sample result likely occurred by random chance or whether it accurately reflects a genuine relationship or difference in the broader population.

Google

Whenever researchers observe a statistical relationship within a sample, they must choose between two mutually exclusive interpretations. The first interpretation asserts that a genuine relationship exists within the broader population, and the sample merely reflects this reality. The alternative interpretation posits that there is no true relationship in the population at all, meaning the observed result in the sample is entirely an artifact of random sampling error.

Two Interpretations of a Statistical Relationship in a Sample

Inferential statistics are a set of mathematical techniques used by researchers to determine whether a statistical result or relationship observed in a study's sample reflects something that is genuinely true of the broader population.

Inferential Statistics

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

The fundamental purpose of null hypothesis testing is to provide researchers with a formal framework to decide between two distinct interpretations of their sample data. It helps determine whether an observed statistical relationship accurately reflects a genuine relationship in the broader population or if it is simply the misleading result of random sampling error.

Purpose of Null Hypothesis Testing

Null Hypothesis Testing

What are the two mutually exclusive interpretations researchers must choose between when they observe a statistical relationship within a sample?

Statistical conclusions in psychological research are fundamentally probabilistic, meaning that empirical findings are always subject to chance and the possibility of statistical errors, such as Type I or Type II errors. Because researchers cannot completely eliminate these potential errors, the results of any single empirical study can never provide absolute certainty or definitively prove that a scientific theory is true.

Probabilistic Nature of Statistics

A finding is considered statistically significant when it is highly unlikely to have occurred due to random chance alone. In the context of inferential statistics, achieving statistical significance suggests that the results represent a genuine, real effect within the broader population.

Statistically Significant

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Type I Error

A Type II error is a false negative in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are not statistically significant—thereby retaining the null hypothesis—when, in reality, the null hypothesis is false and a true relationship does exist in the population. In practical research, Type II errors typically happen because a study lacks adequate statistical power to detect the underlying relationship, often due to an insufficiently large sample size.

Type II Error

In a study of sex differences in talkativeness, Matthias Mehl and his colleagues observed that the women in their sample spoke a mean of $$16,215$$ words per day while men spoke a mean of $$15,669$$ words per day. Despite finding this difference within their specific sample, they ultimately concluded that there was no evidence of a genuine sex difference in talkativeness within the broader population.

Mehl's Study on Sex Differences in Talkativeness

In a study investigating the connection between daily hassles and stress symptoms, Allen Kanner and his colleagues discovered a positive correlation of $$+.60$$ within their sample. Based on this specific sample result, they concluded that a genuine relationship between daily hassles and symptoms exists in the overall population.

Learn Before

Related

Learn After