When a $$p$$ value is greater than the alpha criterion ($$>.05$$), researchers retain the null hypothesis, concluding there is not enough evidence to reject it. Importantly, retaining the null hypothesis is not the same as proving it is true; it simply means the data does not provide sufficient evidence to support the alternative hypothesis. Consequently, researchers use terminology like 'fail to reject the null hypothesis' or 'retain the null hypothesis,' but they never state that they 'accept' the null hypothesis.

Retaining the Null Hypothesis

There is an inherent inverse relationship between the probability of committing a Type I error and a Type II error. If researchers attempt to reduce Type I errors by setting a stricter alpha level (e.g., $$.01$$), they make it harder to reject true null hypotheses, but they inadvertently make it harder to reject false ones, thereby increasing the risk of Type II errors. Conversely, raising the alpha level (e.g., $$.10$$) reduces Type II errors but increases Type I errors. The standard alpha level of $$.05$$ serves as a conventional balance to keep both error rates acceptable.

Trade-off Between Type I and Type II Errors

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Type I Error

In null hypothesis testing, the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis is called ____.

In null hypothesis testing, alpha ($$\alpha$$) is the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis. It is almost always set to $$.05$$. This means that if there is a $$5\%$$ chance or less of observing a result at least as extreme as the sample result when the null hypothesis is true, researchers will reject the null hypothesis.

Google

The p value is the probability of the experiment having given the result it did purely by chance. The threshold of significance is almost always 0.05 so that if the probability of a chance outcome is less than 1 in 20, the experimental result is declared to be significant. There are other important thresholds as summarized in the following table

p value

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

Because the thresholds are conventionalized, we need to recognize the apparent arbitrariness of the conventions and not make such black and white interpretations.

Don't make such black and white interpretations

Always report effect sizes where possible. Effect sizes are measures of importance because they indicate how much effect the experimental manipulation had on the differences seen in the data.

Report effect sizes

The $$p$$-value in null hypothesis testing depends on two primary factors: the strength of the relationship in the sample (effect size) and the size of the sample. A stronger relationship and a larger sample both decrease the likelihood that a result occurred by chance under the null hypothesis, thereby yielding a lower $$p$$-value.

Factors Determining the p-value

A test statistic is a specialized numerical value computed from sample data specifically to aid in determining the $$p$$-value during hypothesis testing. It serves as an intermediate step to evaluate the probability of obtaining the observed results if the null hypothesis were true.

Test Statistic

A common criticism of null hypothesis testing stems from researchers misinterpreting the $$p$$-value. A frequent error is believing that the $$p$$-value represents the probability that the null hypothesis is true, rather than the probability of observing the sample data if the null hypothesis were true. Another closely related misconception is assuming that $$1 - p$$ equals the probability of replicating a statistically significant result, a claim that is overly optimistic and ignores the crucial role of statistical power in replication.

Misinterpretations of the p-value

Alpha (Level of Significance)

p-hacking is a problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result. By selectively reporting dependent variables, unjustified removal of outliers, or selectively presenting only significant findings, researchers continuously manipulate their data until it yields a desirable $$p$$-value. This practice severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors in the psychological literature.

p-hacking

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

The alpha ($$\alpha$$) criterion is a predefined probability threshold—almost always set to $$.05$$ (or $$5\%$$)—that establishes how low a $$p$$ value must be before researchers consider the sample result unlikely enough to reject the null hypothesis.

Alpha Criterion

The most common misinterpretation of the $$p$$ value is mistakenly believing it represents the probability that the null hypothesis is true (i.e., the probability that the result occurred by chance). In reality, the $$p$$ value calculates the probability of obtaining the specific sample data assuming the null hypothesis is already true.

Misinterpretation of the p Value

The $$p$$ value in null hypothesis testing is determined by two main considerations: the strength of the sample relationship and the size of the sample. A stronger sample relationship and a larger sample size both decrease the probability of obtaining the observed result if the null hypothesis were true, resulting in a lower $$p$$ value.

Learn Before

Related

Learn After