Learn Before
Analyze Dr. Smith's analytical decisions in the provided case study. Identify the specific practices she engaged in that constitute p-hacking, and explain the direct consequence these actions have on the published psychological literature.
Case context: Dr. Smith conducts a study on the effect of a new cognitive behavioral therapy on anxiety. She measures anxiety using five different scales (dependent variables). Upon analyzing the data, she finds no significant difference between the treatment and control groups on any scale. She then decides to remove five participants who had very high anxiety scores, stating they were 'distracted.' After removing these outliers, one of the five anxiety scales yields a statistically significant result. Dr. Smith writes her manuscript, focusing entirely on this one significant scale and concluding the therapy is effective. She does not mention the other four scales.
Question: Analyze Dr. Smith's analytical decisions in the provided case study. Identify the specific practices she engaged in that constitute p-hacking, and explain the direct consequence these actions have on the published psychological literature.
Sample answer: Dr. Smith engaged in p-hacking through two specific practices. First, she arbitrarily removed outliers (the five participants) without prior justification, doing so only to manipulate her data until it yielded a desirable -value. Second, she selectively reported only the dependent variable (one anxiety scale) that showed a significant result, while ignoring the four scales that did not. By manipulating her data to achieve a desirable -value, Dr. Smith artificially inflated the chance of obtaining a statistically significant result. This severely compromises the reliability of published results by creating an unacceptably high rate of Type I errors.
Key points:
- Identify the arbitrary and unjustified removal of outliers to manipulate the data.
- Identify the selective reporting of dependent variables and only presenting significant findings.
- Explain that these practices artificially inflate the chances of obtaining a statistically significant result.
- Conclude that this practice compromises reliability by creating an unacceptably high rate of Type I errors in the literature.
Rubric: Full credit requires the student to correctly identify the arbitrary removal of outliers and the selective reporting of dependent variables/significant findings, as well as correctly identifying that these actions artificially inflate the chance of significance and create an unacceptably high rate of Type I errors.
0
1
Tags
KPU
Research Methods in Psychology - 4th American Edition @ KPU
Related
Which of the following best describes the practice of p-hacking in psychological research?
If a researcher conducts multiple analyses on a dataset but selectively publishes only the analysis that yielded a statistically significant result, they are engaging in p-hacking.
A research ethics board is evaluating several study practices for potential bias. Match each p-hacking strategy to the research scenario that demonstrates it.
A researcher is analyzing data from a psychological study and fails to find a statistically significant result. Arrange the following actions in the order they would occur to demonstrate the logical progression of p-hacking to artificially manufacture a significant finding.
You are designing a standardized research protocol for your psychology lab to structurally prevent any opportunity for p-hacking. Based on the mechanisms that enable this practice, which of the following analytical plans must you construct and enforce?
A researcher justifies the selective removal of certain data points only after seeing that their inclusion prevented the study from achieving statistical significance. A reviewer evaluating the scientific integrity of this study would label this practice as _____, as it artificially manipulates the results to obtain a desirable outcome.
The problematic research practice where researchers make various arbitrary analytical decisions to artificially inflate their chances of obtaining a statistically significant result is known as _____.
Imagine you are developing a standardized research protocol for an upcoming psychology experiment. Based on the concept of p-hacking, describe three specific, actionable rules you would implement in your data analysis plan to prevent researchers in your lab from arbitrarily manipulating their data to artificially inflate their chances of obtaining a statistically significant result.
Analyze Dr. Smith's analytical decisions in the provided case study. Identify the specific practices she engaged in that constitute p-hacking, and explain the direct consequence these actions have on the published psychological literature.
A fellow psychology student argues, 'As long as a researcher eventually achieves a desirable -value, the analytical decisions they made along the way—like removing a few outliers or choosing which dependent variables to report—are justified because the significant result proves the phenomenon is real.' Evaluate this argument based on the concept of p-hacking and its impact on statistical error rates.
p-hacking involves making arbitrary analytical decisions to artificially inflate the chances of obtaining a statistically significant result. Match each specific p-hacking practice to its corresponding description.
A researcher establishes a strict mathematical rule for identifying outliers before beginning data collection. After collecting the data, they apply this pre-established rule and remove three participants whose scores fall outside the acceptable range, which subsequently results in a significant -value. This analytical decision is an example of p-hacking.
A researcher conducts an experiment to test a new therapy but engages in p-hacking by selectively reporting dependent variables. Arrange the following events in the correct chronological order to analyze how this specific mechanism of p-hacking unfolds to artificially inflate the chance of a significant finding.