Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6646942/

Power analysis and effect size in mixed effects models: A tutorial

A Type II error is a false negative in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are not statistically significant—thereby retaining the null hypothesis—when, in reality, the null hypothesis is false and a true relationship does exist in the population. In practical research, Type II errors typically happen because a study lacks adequate statistical power to detect the underlying relationship, often due to an insufficiently large sample size.

Type II Error

If a research design lacks adequate statistical power, researchers can increase it by either increasing the sample size or strengthening the relationship. Strengthening the relationship can be achieved by using a stronger experimental manipulation or by carefully controlling extraneous variables to reduce data noise, such as employing a within-subjects design instead of a between-subjects design. However, the most common and practical strategy to boost power is simply to increase the sample size.

Increasing Statistical Power

Because manual computation is complex, researchers typically compute the statistical power for various research designs using online tools and downloadable programs, such as Russ Lenth’s Power and Sample Size Page or G*Power. By inputting the expected relationship strength, the sample size, and the $$\alpha$$ level, these applications calculate the study's power or determine the exact sample size required to achieve a desired power level, such as $$.80$$.

Computing Statistical Power Online

Statistical power and the probability of committing a Type II error are perfectly complementary. A Type II error occurs when a researcher fails to reject a false null hypothesis (missing a real effect). Since statistical power is the probability of correctly rejecting that false null hypothesis, statistical power is calculated as $$1 - 	ext{probability of a Type II error}$$. Therefore, maximizing a study's statistical power is the primary method for minimizing the risk of making a Type II error.

Relationship Between Statistical Power and Type II Error

In psychological research, it is generally recommended that a study's design possess adequate power prior to data collection. A widely accepted guideline is that an adequate level of statistical power is $$.80$$. This indicates that if the expected effect or relationship truly exists in the population, the study has an $$80\%$$ chance of successfully detecting it and rejecting the null hypothesis.

Adequate Statistical Power

To achieve high statistical power in psychological research, sample sizes must be larger than often realized. For an independent-samples $$t$$-test with a large population effect, it requires approximately $$26$$ participants per sample to reach a power of $$0.80$$, while reaching a power of $$0.99$$ requires $$59$$ participants per sample.

Example of Sample Size Requirements for High Statistical Power

Statistical power is heavily dependent on the expected relationship strength (effect size). Weaker relationships are harder to detect and thus provide lower statistical power for a given sample size. To compensate and achieve adequate power when investigating a weak relationship, researchers must utilize very large sample sizes.

Relationship Between Statistical Power and Relationship Strength

Which of the following best defines the statistical power of a research design?

A researcher designs a study with a statistical power of .80. This means there is a 20% probability that the study will fail to detect a real effect that actually exists in the population.

The statistical power of a research design is the probability that the study will reject the null hypothesis, given a specific sample size and an expected relationship strength. For instance, if a design has a power of $$.80$$, it means there is an $$80\%$$ chance of correctly rejecting the null hypothesis when the expected relationship exists in the population. It is essentially the complement of the probability of making a Type II error, so researchers strive for high statistical power to avoid missing a real effect.

Google

University of Michigan - Ann Arbor

Components of Statistical Analysis

Statement that says there is no relationship between the two measured items

Null Hypothesis

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

Statistical Power

A quantitative measure, such as Cohen's $$d$$ or Pearson's $$r$$, used to estimate the strength of a statistical relationship within a population. In response to criticisms of null hypothesis testing, researchers are encouraged to report effect sizes alongside their tests. This practice is crucial because a $$p$$-value alone only indicates whether a relationship is statistically significant—which is heavily influenced by sample size—but fails to convey the actual magnitude or importance of that relationship.

Effect Size

Critics of null hypothesis testing argue that the strict convention of rejecting the null hypothesis only when $$p < .05$$ is logically flawed. This rigid threshold creates an artificial divide where a result of $$p = .04$$ is deemed significant and worthy of publication, while a nearly identical result of $$p = .06$$ is dismissed as not significant. This arbitrariness can prevent valuable research from being published and exacerbates the file drawer problem by suppressing non-significant findings.

Arbitrariness of the p-value Threshold

Bayesian statistics is an approach to inferential statistics where a researcher specifies the prior probability that the null hypothesis and alternative hypotheses are true before conducting a study. Once the study is conducted, the probabilities are updated based on the newly collected data. This approach offers a potential alternative to traditional null hypothesis testing by providing a logical framework for quantifying uncertainty using probability distributions.

Bayesian Statistics

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Learn Before

Related

Learn After