Learn Before
Based on the definition of a value and a significance level of set in advance, explain in your own words what it means if, assuming the null hypothesis and the test's assumptions were true, a result at least as extreme as the researchers' would occur less than 1 time in 20, and what this does not establish.
Case context: A group of researchers conducts an experiment comparing a new memory recall technique with a standard one. Before analyzing their data, they set a significance level of ; after the analysis, they report a value for the difference between the two conditions. They must now explain to a non-scientific audience what the reported value means, how it compares with their stated threshold, and what neither the value nor the threshold can establish on its own.
Question: Based on the definition of a value and a significance level of set in advance, explain in your own words what it means if, assuming the null hypothesis and the test's assumptions were true, a result at least as extreme as the researchers' would occur less than 1 time in 20, and what this does not establish.
Sample answer: It means the value is less than : if the new technique made no difference (the null hypothesis) and the test's assumptions held, a difference at least as large as the one observed would occur less than 1 time in 20. Because this is below the stated significance level of , the researchers would call the result statistically significant. This does not mean there is less than a 1 in 20 chance that the result was due to chance, or that the null hypothesis is probably false. On its own, it also does not establish that the technique works, how large its effect is, or whether the effect matters in practice; that depends on the effect size, the study's design and other evidence.
Key points:
- Less than 1 in 20 under the null hypothesis and the test's assumptions corresponds to .
- Compared with the stated significance level of , the result is called statistically significant.
- The value is not the probability that the result is due to chance or that the null hypothesis is true.
- Neither the value nor the threshold alone establishes that the technique works, how large its effect is, or whether it matters.
Rubric: Award full points if the student (1) connects "less than 1 in 20, assuming the null hypothesis and the test's assumptions" to ; (2) states that the result is statistically significant at the stated level; (3) does not describe the value as the probability that the result is due to chance or that the null hypothesis is true; and (4) notes that significance alone does not establish that the technique works, how large its effect is, or whether it matters.
0
1
Tags
KPU
Research Methods in Psychology - 4th American Edition @ KPU
Related
Don't make such black and white interpretations
Factors Determining the p-value
Test Statistic
Misinterpretations of the p-value
Alpha (Level of Significance)
p-hacking
The 2015 Ban on Null Hypothesis Testing
Alpha Criterion
What does a p value represent in the context of psychological research?
If a psychology researcher calculates a p-value of 0.08, they will typically declare the experimental result to be statistically significant.
A social psychologist is analyzing data from four different studies. Based on the standard thresholds for statistical significance provided in the table, match each study's obtained p value with its correct interpretation.
Misinterpretation of the p Value
Determinants of the Value
A researcher obtains a value of and labels the result 'highly significant' (). According to standard significance thresholds, where and , this labeling is _____.
A cognitive psychologist conducts an experiment on learning styles and calculates a value of . Applying the standard threshold of significance () described in the text, the researcher should declare this result to be statistically significant.
Define what a value represents in the context of an experiment and state the standard threshold for declaring an experimental result to be statistically significant.
Reporting Effect Sizes in Null Hypothesis Significance Testing
Using the convention stated in this question for marking results with asterisks (a cut-off value itself counts as meeting it), match each label with the range of values it marks.
By a common convention in psychology, a result is declared statistically significant when . For a result exactly at this threshold (), results at least as extreme would occur, under the test's model (the null hypothesis plus the test's assumptions), time in how many?
By a common convention in psychology, a result is declared statistically significant when . For a result exactly at this threshold (), if the null hypothesis and the test's assumptions were true, results at least as extreme would occur 1 time in _____.
Arrange the following three values in ascending numerical order, from the smallest (1) to the largest (3).
A psychologist has values from four different studies. Arrange the values in ascending numerical order, from the smallest to the largest. (This orders the numbers only: values from different studies, each calculated under its own model, do not by themselves rank how strong the evidence is.)
A psychology student runs an analysis on their study's data and obtains a value of . Using the common convention of a significance level of (a result is significant when ), what conclusion should the student draw about their experimental result, and why?
Based on the definition of a value and a significance level of set in advance, explain in your own words what it means if, assuming the null hypothesis and the test's assumptions were true, a result at least as extreme as the researchers' would occur less than 1 time in 20, and what this does not establish.