Students are asked an open-ended question relating to a topic of interest. Experts on this topic are then given two student responses, and they judge which of the two displays a better conceptual understanding of the concept without using a specified scoring rubric. This process is repeated several times, and a rank/scale is created for the responses from “worst” to “best.” Using statistical modeling, a z score parameter and standard error is created for the responses. Studies have found that comparative judgement displays internal consistency, reliability, and validity in varying (mathematical) contexts. 

Comparative Judgement Process

- It is argued that CJ is too “under defined;” however, when discussing open ended questions, specified scoring procedures can be too rigid and may discount the nuances in responses that show understanding. 
- Some say CJ is too subjective, but when multiple experts review responses, biases can be cancelled out.
- It is hard to definitively say that CJ is only measuring conceptual understanding and not procedural knowledge.
- Operationalizing conceptual and procedural knowledge may also come to question.


Arguments Against Comparative Judgement

A method used to measure conceptual understanding (usually in mathematics) that does not require traditional scoring rubrics. Comparative judgement is based off the psychological principle “that humans are better at comparing two objects against one another than they are at comparing one object against specified criteria.”

Michigan State University

Understanding the deep relationships between pieces of information and the principles of their domain

Conceptual Understanding

Bisson, MJ., Gilmore, C., Inglis, M. et al. Measuring Conceptual Understanding Using Comparative Judgement. Int. J. Res. Undergrad. Math. Ed. 2, 141–164 (2016). https://doi.org/10.1007/s40753-016-0024-3

Learn Before

Related

Learn After