Learn Before
Reliability’s two taxonomy levels in COSMIN
In COSMIN’s taxonomy for health-related patient-reported outcome (HR-PRO) instruments, reliability appears at two levels. As a domain, it concerns freedom from measurement error. Its extended definition asks whether scores remain the same when patients have not changed and measurements repeat under specified conditions.
As a measurement property, reliability is the proportion of total measurement variance attributable to true differences between patients. In the table’s classical test theory footnote, a true score is the average score from infinitely many repetitions using the same instrument. The footnote uses a consistency definition of true score; validity separately addresses measurement of the intended construct.
The extended domain definition lists internal consistency, test-retest, inter-rater and intra-rater measurement contexts. Measurement error is a separate property concerning systematic and random score error unrelated to real changes in the construct. Reading the table column as well as the label prevents treating broad reproducibility and the variance-based property as interchangeable definitions.
0
1
Tags
Social Science
Empirical Science
Science
Ch.1 Introduction to Psychology - Psychology @ OpenStax
OpenStax
Psychology @ OpenStax
Ch.2 Psychological Research - Psychology @ OpenStax
Introduction to Psychology @ OpenStax Course
OpenStax Psychology (2nd ed.) Textbook
Psychology
KPU
Research Methods in Psychology - 4th American Edition @ KPU
Ch.7 Thinking and Intelligence - Psychology @ OpenStax
Related
Types of Reliability
A team of researchers develops a new questionnaire designed to measure an individual's level of creativity. Which of the following outcomes would provide the strongest evidence that the new questionnaire is reliable?
A research team develops a new observational checklist to measure 'attentive behavior' in preschoolers. Two different researchers use the checklist to observe the same child at the same time, but their final scores for the child's attentiveness are completely different. When this process is repeated with other children, the two researchers' scores continue to show no relationship to each other. Based on this information, what is the most significant problem with this new checklist?
Replication of Studies in Psychology
Alfred Binet's Intelligence Test
Standardization
In psychological research, what does the reliability of a measurement tool refer to?
A researcher administers a newly developed intelligence test to a group of students and then re-administers it to the same group a week later under the same conditions. If the students' scores are significantly different between the two administrations, the test is demonstrating high reliability.
In psychological research, reliability manifests in several ways. Match each research scenario with the specific type of consistency (or lack thereof) it demonstrates.
A research team is evaluating the reliability of a new 'Social Anxiety Scale' to determine if it is stable enough for use in a long-term clinical study. Arrange the following findings in order, from the result that provides the least support for the tool's consistency to the result that provides the most robust evidence for its stability over time.
In the context of psychological research, what is the primary characteristic of a 'reliable' measurement tool, such as an intelligence assessment?
In psychological research, reliability involves maintaining consistency under different conditions. Match each term with the scenario or description that best illustrates it.
A researcher is evaluating a new intelligence assessment. A student takes the test on Monday and scores . On Tuesday, under identical conditions and with no changes in the student's cognitive state, the student takes the test again and scores . By analyzing the discrepancy between these two results for a stable trait, the researcher concludes that the assessment lacks _____.
A clinical psychologist administers a new spatial reasoning test to a group of participants on Monday, and then administers the exact same test to the same participants on Friday under identical conditions. If the participants receive almost identical scores on both administrations, this outcome demonstrates that the spatial reasoning test is reliable.
A research team evaluates a new diagnostic checklist. When Psychologist A uses the checklist to evaluate a patient, they diagnose them with mild depression, but when Psychologist B evaluates the same patient on the same day using the same checklist, they diagnose them with severe anxiety. Analyzing these discrepant outcomes suggests that the diagnostic checklist lacks _____.
A psychologist is evaluating the results of test-retest administrations for four experimental intelligence assessment tools. Evaluate the strength of the reliability evidence based on their Pearson correlation coefficients (), and arrange the assessments in order from the one showing the STRONGEST evidence of reliability to the one showing the WEAKEST evidence of reliability.
In psychological research, what does the term 'reliability' refer to when evaluating a measurement tool?
If a newly developed intelligence test yields drastically different scores each time the same person takes it under identical conditions, the test demonstrates high reliability.
A cognitive psychologist uses a computerized spatial reasoning task to measure participants' baseline skills. If a participant takes the exact same task three times in one week under identical laboratory conditions and receives vastly different scores each time, the researcher should conclude that this measurement tool lacks ____.
Analyze the following psychological research scenarios and match each to the specific aspect of measurement reliability it best demonstrates.
As a peer reviewer, you are tasked with evaluating the reliability of a new psychological measurement tool introduced in a research paper. Arrange the steps of your critical evaluation process in the correct logical order, from initial identification to final judgment.
In psychological research, a measurement tool's ability to produce consistent and stable results over time or across different situations is defined as its validity.
A psychologist is developing a new self-report questionnaire to measure trait anxiety. If the psychologist wants to establish the reliability of this questionnaire, what must they demonstrate?
A psychology researcher wants to verify that a newly developed intelligence test produces consistent and stable results over time. Arrange the practical steps the researcher must take to assess the reliability of this measurement tool.
A psychologist is analyzing data from a pilot study evaluating a new intelligence test. The data reveals that participants receive drastically different scores depending on whether the test is administered in a quiet laboratory or a busy classroom, even though their actual cognitive abilities remain constant. Based on the provided definition of reliability, how should the psychologist critically interpret this specific finding?
As a peer reviewer, you must critically evaluate the measurement tools used in various psychological studies. Match each research scenario to the most appropriate evaluative judgment regarding the tool's reliability.
Reliability’s two taxonomy levels in COSMIN