In null hypothesis testing, sample size and relationship strength trade off against each other to determine statistical significance. A weak statistical relationship can still reach statistical significance if the sample size is sufficiently large, and conversely, a very strong relationship can be statistically significant even if it is based on a small sample.

Trade-off Between Sample Size and Relationship Strength

Which two primary factors determine the $$p$$-value in null hypothesis testing?

The $$p$$-value in null hypothesis testing depends on two primary factors: the strength of the relationship in the sample (effect size) and the size of the sample. A stronger relationship and a larger sample both decrease the likelihood that a result occurred by chance under the null hypothesis, thereby yielding a lower $$p$$-value.

Google

The p value is the probability of the experiment having given the result it did purely by chance. The threshold of significance is almost always 0.05 so that if the probability of a chance outcome is less than 1 in 20, the experimental result is declared to be significant. There are other important thresholds as summarized in the following table

p value

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

Because the thresholds are conventionalized, we need to recognize the apparent arbitrariness of the conventions and not make such black and white interpretations.

Don't make such black and white interpretations

Always report effect sizes where possible. Effect sizes are measures of importance because they indicate how much effect the experimental manipulation had on the differences seen in the data.

Report effect sizes

Factors Determining the p-value

A test statistic is a specialized numerical value computed from sample data specifically to aid in determining the $$p$$-value during hypothesis testing. It serves as an intermediate step to evaluate the probability of obtaining the observed results if the null hypothesis were true.

Test Statistic

A common criticism of null hypothesis testing stems from researchers misinterpreting the $$p$$-value. A frequent error is believing that the $$p$$-value represents the probability that the null hypothesis is true, rather than the probability of observing the sample data if the null hypothesis were true. Another closely related misconception is assuming that $$1 - p$$ equals the probability of replicating a statistically significant result, a claim that is overly optimistic and ignores the crucial role of statistical power in replication.

Misinterpretations of the p-value

In null hypothesis testing, alpha ($$\alpha$$) is the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis. It is almost always set to $$.05$$. This means that if there is a $$5\%$$ chance or less of observing a result at least as extreme as the sample result when the null hypothesis is true, researchers will reject the null hypothesis.

Alpha (Level of Significance)

p-hacking is a problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result. By selectively reporting dependent variables, unjustified removal of outliers, or selectively presenting only significant findings, researchers continuously manipulate their data until it yields a desirable $$p$$-value. This practice severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors in the psychological literature.

p-hacking

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

The alpha ($$\alpha$$) criterion is a predefined probability threshold—almost always set to $$.05$$ (or $$5\%$$)—that establishes how low a $$p$$ value must be before researchers consider the sample result unlikely enough to reject the null hypothesis.

Alpha Criterion

The most common misinterpretation of the $$p$$ value is mistakenly believing it represents the probability that the null hypothesis is true (i.e., the probability that the result occurred by chance). In reality, the $$p$$ value calculates the probability of obtaining the specific sample data assuming the null hypothesis is already true.

Misinterpretation of the p Value

The $$p$$ value in null hypothesis testing is determined by two main considerations: the strength of the sample relationship and the size of the sample. A stronger sample relationship and a larger sample size both decrease the probability of obtaining the observed result if the null hypothesis were true, resulting in a lower $$p$$ value.

Learn Before

Related

Learn After