Don't make such black and white interpretations

Factors Determining the p-value

Test Statistic

Misinterpretations of the p-value

Alpha (Level of Significance)

p-hacking

The 2015 Ban on Null Hypothesis Testing

Alpha Criterion

What does a p value represent in the context of psychological research?

If a psychology researcher calculates a p-value of 0.08, they will typically declare the experimental result to be statistically significant.

A social psychologist is analyzing data from four different studies. Based on the standard thresholds for statistical significance provided in the table, match each study's obtained p value with its correct interpretation.

A psychologist is comparing findings across four different research studies. Based on the definition of a p value and the significance thresholds provided in the table, arrange these findings in order from the outcome LEAST likely to have occurred by chance to the outcome MOST likely to have occurred by chance.

According to the standard threshold of significance ($p = 0.05$) in psychology, a research result is typically declared significant if the probability of it occurring purely by chance is less than $1$ in how many?

Misinterpretation of the p Value

Determinants of the $p$ Value

Based on the significance thresholds provided in the table and the definition of a $p$ value, match each statistical label with the correct description of its probability occurring purely by chance.

A researcher obtains a $p$ value of 0.02 and labels the result 'highly significant' ($**$). According to standard significance thresholds, where $* = p \leq 0.05$ and $** = p \leq 0.01$, this labeling is _____.

A cognitive psychologist conducts an experiment on learning styles and calculates a $p$ value of 0.04. Applying the standard threshold of significance ($p = 0.05$) described in the text, the researcher should declare this result to be statistically significant.

According to the standard threshold of significance ($p = 0.05$), a psychological research result is typically declared significant if the probability of the outcome occurring purely by chance is less than 1 in _____.

Evaluate the following three experimental results and arrange them in order from the outcome representing the lowest probability of occurring purely by chance (1) to the outcome representing the highest probability of occurring purely by chance (3).

Define what a $p$ value represents in the context of an experiment and state the standard threshold for declaring an experimental result to be statistically significant.

Based on the standard definition of a $p$ value and its typical threshold, explain in your own words what it means if the researchers determined the probability of their outcome occurring purely by chance was less than 1 in 20.

A psychology student runs an analysis on their study's data and discovers the probability of their results occurring purely by chance is 0.03. Applying the standard threshold of significance, what conclusion should the student draw about their experimental result, and why?

Reporting Effect Sizes in Null Hypothesis Significance Testing

Effect Size in Studies Lacking Power

Correlation Coefficient

What does an effect size estimate in a statistical analysis?

Even when a relationship is found to be statistically significant, an effect size should be reported because a p-value alone cannot convey the actual magnitude or importance of that relationship.

Match each hypothetical research outcome with the interpretation that accurately describes its practical and statistical significance based on the reported effect size and p-value.

A psychology researcher reports the following statistical outcomes from four separate studies. Arrange these results in order from the weakest relationship magnitude to the strongest relationship magnitude in the population, regardless of their statistical significance levels.

A psychology researcher is synthesizing a formal 'Discussion' section for a study that found a statistically significant difference between groups ($p = .04$) but a very low magnitude of relationship ($d = 0.10$). Arrange the following statements in the correct logical sequence to construct a scientifically rigorous interpretation that prioritizes the actual strength of the finding over its probability.

Why are psychology researchers strongly encouraged to report an effect size (such as Cohen's $d$ or Pearson's $r$) in addition to a $p$-value from a null hypothesis significance test?

A quantitative measure, such as Cohen's $d$ or Pearson's $r$, that is used to estimate the strength of a statistical relationship within a population is known as an _____.

A researcher is evaluating two separate studies to decide which memory-enhancement technique to recommend for a clinical trial. Study 1 reports a significant result with $p < .001$ and an effect size of $d = 0.12$. Study 2 reports a significant result with $p = .04$ and an effect size of $d = 0.82$. If the researcher evaluates these findings based on the strength of the relationship in the population rather than just the probability of the results occurring by chance, they should conclude that Study _____ provides more compelling evidence of a powerful technique.

A researcher runs two independent studies on the same phenomenon. Study A ($n = 500$) yields $p = .04$ and $d = 0.09$. Study B ($n = 25$) yields $p = .08$ and $d = 0.63$. Analyzing only the statistical significance of these outcomes, a peer reviewer would correctly conclude that Study A provides stronger evidence for a meaningful population-level relationship than Study B.

A journal reviewer is evaluating four researcher statements about statistical results. Match each statement to the critique that best assesses its reporting quality.

Based on the provided statistical guidelines, define "effect size" and list the two specific quantitative measures of effect size mentioned in the text. Finally, state why researchers are encouraged to report these measures alongside null hypothesis tests.

Explain why the researcher's conclusion about the importance and power of the study aid is misleading. In your explanation, describe how the sample size of 15,000 students affected the $p$-value, and explain what Cohen's $d = 0.03$ reveals about the actual magnitude of the relationship.

You are writing the discussion section for a psychology experiment with a sample size of 40 participants. You found a statistically significant result ($p = .035$). Apply the statistical reporting recommendations from the text to justify why you should report an effect size measure (such as Pearson's $r$) alongside this $p$-value.

Cohen's $d$

Reporting Effect Sizes in Null Hypothesis Significance Testing