A standardized test development committee pilots two new questions. On Question X, students from historically high-scoring demographic groups outperform students from historically low-scoring groups. On Question Y, the opposite occurs. The committee decides to include Question X and discard Question Y, arguing that a question is 'reliable' only if its results are consistent with the test's historical performance patterns. What is the most significant flaw in the committee's reasoning?
0
1
Tags
Disability Studies
Ch.2 Psychological Research - Psychology @ OpenStax
Psychology @ OpenStax
Introduction to Psychology @ OpenStax Course
Social Science
Empirical Science
Science
OpenStax
OpenStax Psychology (2nd ed.) Textbook
Psychology
Analysis in Bloom's Taxonomy
Cognitive Psychology
Related
A standardized test development committee pilots two new questions. On Question X, students from historically high-scoring demographic groups outperform students from historically low-scoring groups. On Question Y, the opposite occurs. The committee decides to include Question X and discard Question Y, arguing that a question is 'reliable' only if its results are consistent with the test's historical performance patterns. What is the most significant flaw in the committee's reasoning?
A standardized test development agency pilots two new questions. The agency's goal is to select questions that are 'reliable,' meaning they produce results consistent with historical test data where Group X has consistently outscored Group Y. Below are the results for the two pilot questions:
- Question 1: 70% of students from Group X answered correctly, while 40% of students from Group Y answered correctly.
- Question 2: 55% of students from Group X answered correctly, while 65% of students from Group Y answered correctly.
Based on the agency's validation process, which outcome is most likely?