Which of the following best describes the practice of p-hacking in psychological research?

p-hacking is a problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result. By selectively reporting dependent variables, unjustified removal of outliers, or selectively presenting only significant findings, researchers continuously manipulate their data until it yields a desirable $$p$$-value. This practice severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors in the psychological literature.

Google

The p value is the probability of the experiment having given the result it did purely by chance. The threshold of significance is almost always 0.05 so that if the probability of a chance outcome is less than 1 in 20, the experimental result is declared to be significant. There are other important thresholds as summarized in the following table

p value

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Type I Error

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

Because the thresholds are conventionalized, we need to recognize the apparent arbitrariness of the conventions and not make such black and white interpretations.

Don't make such black and white interpretations

Always report effect sizes where possible. Effect sizes are measures of importance because they indicate how much effect the experimental manipulation had on the differences seen in the data.

Report effect sizes

The $$p$$-value in null hypothesis testing depends on two primary factors: the strength of the relationship in the sample (effect size) and the size of the sample. A stronger relationship and a larger sample both decrease the likelihood that a result occurred by chance under the null hypothesis, thereby yielding a lower $$p$$-value.

Factors Determining the p-value

A test statistic is a specialized numerical value computed from sample data specifically to aid in determining the $$p$$-value during hypothesis testing. It serves as an intermediate step to evaluate the probability of obtaining the observed results if the null hypothesis were true.

Test Statistic

A common criticism of null hypothesis testing stems from researchers misinterpreting the $$p$$-value. A frequent error is believing that the $$p$$-value represents the probability that the null hypothesis is true, rather than the probability of observing the sample data if the null hypothesis were true. Another closely related misconception is assuming that $$1 - p$$ equals the probability of replicating a statistically significant result, a claim that is overly optimistic and ignores the crucial role of statistical power in replication.

Misinterpretations of the p-value

In null hypothesis testing, alpha ($$\alpha$$) is the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis. It is almost always set to $$.05$$. This means that if there is a $$5\%$$ chance or less of observing a result at least as extreme as the sample result when the null hypothesis is true, researchers will reject the null hypothesis.

Alpha (Level of Significance)

p-hacking

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

The alpha ($$\alpha$$) criterion is a predefined probability threshold—almost always set to $$.05$$ (or $$5\%$$)—that establishes how low a $$p$$ value must be before researchers consider the sample result unlikely enough to reject the null hypothesis.

Alpha Criterion

The most common misinterpretation of the $$p$$ value is mistakenly believing it represents the probability that the null hypothesis is true (i.e., the probability that the result occurred by chance). In reality, the $$p$$ value calculates the probability of obtaining the specific sample data assuming the null hypothesis is already true.

Misinterpretation of the p Value

The $$p$$ value in null hypothesis testing is determined by two main considerations: the strength of the sample relationship and the size of the sample. A stronger sample relationship and a larger sample size both decrease the probability of obtaining the observed result if the null hypothesis were true, resulting in a lower $$p$$ value.

Determinants of the $$p$$ Value

What does a p value represent in the context of psychological research?

The file drawer problem refers to a systemic publication bias where researchers and journal editors selectively publish studies that yield statistically significant results while ignoring or discarding studies with non-significant findings. Because these non-significant results are hidden away 'in a file drawer,' the published scientific literature becomes skewed, often containing a disproportionately high rate of Type I errors. Consequently, the published research may artificially overstate the true strength and prevalence of relationships between variables in the population.

File Drawer Problem

In psychological research, replication involves repeating another scientist's experiment, often using new samples, to assess the reliability of the original findings. Because any single study carries a risk of reflecting a Type I or Type II error, researchers must be cautious when interpreting isolated results. Each successful replication of a study that produces a similar outcome provides increasing confidence that the finding represents a genuine, real-world phenomenon rather than a statistical artifact.

Replication of Studies in Psychology

A memorable illustration of statistical errors involves medical diagnoses. A Type I error (a false positive) is akin to a doctor examining a male patient and incorrectly declaring, 'You are pregnant.' The doctor has claimed to detect a condition that does not actually exist. Conversely, a Type II error (a false negative) is akin to a doctor examining a visibly pregnant female patient and incorrectly concluding, 'You are not pregnant.' In this case, the doctor has failed to detect a condition that is genuinely present.

Learn Before

Related

Learn After