The file drawer problem refers to a systemic publication bias where researchers and journal editors selectively publish studies that yield statistically significant results while ignoring or discarding studies with non-significant findings. Because these non-significant results are hidden away 'in a file drawer,' the published scientific literature becomes skewed, often containing a disproportionately high rate of Type I errors. Consequently, the published research may artificially overstate the true strength and prevalence of relationships between variables in the population.

File Drawer Problem

p-hacking is a problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result. By selectively reporting dependent variables, unjustified removal of outliers, or selectively presenting only significant findings, researchers continuously manipulate their data until it yields a desirable $$p$$-value. This practice severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors in the psychological literature.

p-hacking

In psychological research, replication involves repeating another scientist's experiment, often using new samples, to assess the reliability of the original findings. Because any single study carries a risk of reflecting a Type I or Type II error, researchers must be cautious when interpreting isolated results. Each successful replication of a study that produces a similar outcome provides increasing confidence that the finding represents a genuine, real-world phenomenon rather than a statistical artifact.

Replication of Studies in Psychology

A memorable illustration of statistical errors involves medical diagnoses. A Type I error (a false positive) is akin to a doctor examining a male patient and incorrectly declaring, 'You are pregnant.' The doctor has claimed to detect a condition that does not actually exist. Conversely, a Type II error (a false negative) is akin to a doctor examining a visibly pregnant female patient and incorrectly concluding, 'You are not pregnant.' In this case, the doctor has failed to detect a condition that is genuinely present.

Example of Type I and Type II Errors

In null hypothesis testing, which of the following best defines a Type I error?

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Google

Statement that says there is no relationship between the two measured items

Null Hypothesis

In the logic of null hypothesis testing, researchers reject the null hypothesis in favor of the alternative hypothesis if they determine that the observed sample relationship would be extremely unlikely to occur if the null hypothesis were actually true. This formal decision indicates that the sample data provides sufficient statistical evidence to conclude that a genuine relationship exists within the broader population.

Rejecting the Null Hypothesis

In null hypothesis testing, alpha ($$\alpha$$) is the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis. It is almost always set to $$.05$$. This means that if there is a $$5\%$$ chance or less of observing a result at least as extreme as the sample result when the null hypothesis is true, researchers will reject the null hypothesis.

Alpha (Level of Significance)

Inferential statistics are a set of mathematical techniques used by researchers to determine whether a statistical result or relationship observed in a study's sample reflects something that is genuinely true of the broader population.

Inferential Statistics

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

The statistical power of a research design is the probability that the study will reject the null hypothesis, given a specific sample size and an expected relationship strength. For instance, if a design has a power of $$.80$$, it means there is an $$80\%$$ chance of correctly rejecting the null hypothesis when the expected relationship exists in the population. It is essentially the complement of the probability of making a Type II error, so researchers strive for high statistical power to avoid missing a real effect.

Statistical Power

Critics of null hypothesis testing argue that the strict convention of rejecting the null hypothesis only when $$p < .05$$ is logically flawed. This rigid threshold creates an artificial divide where a result of $$p = .04$$ is deemed significant and worthy of publication, while a nearly identical result of $$p = .06$$ is dismissed as not significant. This arbitrariness can prevent valuable research from being published and exacerbates the file drawer problem by suppressing non-significant findings.

Arbitrariness of the p-value Threshold

Bayesian statistics is an approach to inferential statistics where a researcher specifies the prior probability that the null hypothesis and alternative hypotheses are true before conducting a study. Once the study is conducted, the probabilities are updated based on the newly collected data. This approach offers a potential alternative to traditional null hypothesis testing by providing a logical framework for quantifying uncertainty using probability distributions.

Bayesian Statistics

Type I Error

A Type II error is a false negative in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are not statistically significant—thereby retaining the null hypothesis—when, in reality, the null hypothesis is false and a true relationship does exist in the population. In practical research, Type II errors typically happen because a study lacks adequate statistical power to detect the underlying relationship, often due to an insufficiently large sample size.

Type II Error

In the context of psychological research, which statement best defines a null hypothesis?

In the logic of null hypothesis testing, what formal decision indicates that the sample data provides sufficient statistical evidence to conclude that a genuine relationship exists within the broader population?

When a $$p$$ value is greater than the alpha criterion ($$>.05$$), researchers retain the null hypothesis, concluding there is not enough evidence to reject it. Importantly, retaining the null hypothesis is not the same as proving it is true; it simply means the data does not provide sufficient evidence to support the alternative hypothesis. Consequently, researchers use terminology like 'fail to reject the null hypothesis' or 'retain the null hypothesis,' but they never state that they 'accept' the null hypothesis.

Retaining the Null Hypothesis

There is an inherent inverse relationship between the probability of committing a Type I error and a Type II error. If researchers attempt to reduce Type I errors by setting a stricter alpha level (e.g., $$.01$$), they make it harder to reject true null hypotheses, but they inadvertently make it harder to reject false ones, thereby increasing the risk of Type II errors. Conversely, raising the alpha level (e.g., $$.10$$) reduces Type II errors but increases Type I errors. The standard alpha level of $$.05$$ serves as a conventional balance to keep both error rates acceptable.

Trade-off Between Type I and Type II Errors

In null hypothesis testing, the predetermined criterion used to decide how low a $$p$$ value must be before a sample result is considered unlikely enough to reject the null hypothesis is called ____.

Statistical conclusions in psychological research are fundamentally probabilistic, meaning that empirical findings are always subject to chance and the possibility of statistical errors, such as Type I or Type II errors. Because researchers cannot completely eliminate these potential errors, the results of any single empirical study can never provide absolute certainty or definitively prove that a scientific theory is true.

Probabilistic Nature of Statistics

A statistical range calculated around a sample statistic designed to contain the true population parameter a specified percentage of the time, most commonly $$95\%$$. Some researchers advocate using confidence intervals as an alternative to traditional null hypothesis testing because they are often more intuitive to interpret. Furthermore, a confidence interval inherently provides the data needed to perform a null hypothesis test; if a hypothetical population mean falls completely outside the calculated interval, the sample mean is considered statistically significantly different from that hypothetical value.

Confidence Interval

A finding is considered statistically significant when it is highly unlikely to have occurred due to random chance alone. In the context of inferential statistics, achieving statistical significance suggests that the results represent a genuine, real effect within the broader population.

Statistically Significant

In a study of sex differences in talkativeness, Matthias Mehl and his colleagues observed that the women in their sample spoke a mean of $$16,215$$ words per day while men spoke a mean of $$15,669$$ words per day. Despite finding this difference within their specific sample, they ultimately concluded that there was no evidence of a genuine sex difference in talkativeness within the broader population.

Mehl's Study on Sex Differences in Talkativeness

In a study investigating the connection between daily hassles and stress symptoms, Allen Kanner and his colleagues discovered a positive correlation of $$+.60$$ within their sample. Based on this specific sample result, they concluded that a genuine relationship between daily hassles and symptoms exists in the overall population.

Kanner's Study on Daily Hassles and Symptoms

Null hypothesis testing is the most common approach to inferential statistics in psychological research. It provides researchers with a formal, structured process to evaluate whether an observed sample result likely occurred by random chance or whether it accurately reflects a genuine relationship or difference in the broader population.

Learn Before

Related

Learn After