A researcher tests a new therapy designed to reduce anxiety. After the study, a statistical analysis indicates there is a 4% probability that the observed reduction in anxiety in the therapy group occurred purely by chance. Based on the standard conventions for drawing conclusions from data, should the researcher claim that the therapy has a genuine effect? Justify your answer.

Interpreting a Study's Outcome

A primary goal in statistical analysis is to limit the rate of 'false positive' claims, which are erroneous conclusions that an effect exists when it does not. Psychologists achieve this by setting a strict criterion for statistical significance, commonly accepting a 5% or lower probability that the observed results are due to chance.

Google

In psychological research, a finding is typically considered 'statistically significant' if there is a probability of five percent or less that the observed difference between groups occurred by random chance. This standard helps researchers confidently conclude that the results are meaningful and not just a fluke.

Statistical Significance in Psychology

Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2020, April 22). Psychology (2nd ed.). OpenStax. https://openstax.org/books/psychology-2e/pages/1-introduction

OpenStax Psychology (2nd ed.) Textbook

Limiting False Positives in Research

If an experiment evaluates a nutritional supplement and finds that participants taking a placebo achieve the same outcomes as those taking the active supplement, the difference is not meaningful. In this scenario, statistical analysis would show the result is likely due to chance, leading to the conclusion that the supplement is ineffective.

Example of a Non-Significant Result: Nutritional Supplement

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Criticisms of Null Hypothesis Testing

In psychological research, a finding is typically considered 'statistically significant' if the probability that the observed difference occurred by random chance is:

Learn Before

Related

Learn After