- It is argued that CJ is too “under defined;” however, when discussing open ended questions, specified scoring procedures can be too rigid and may discount the nuances in responses that show understanding. 
- Some say CJ is too subjective, but when multiple experts review responses, biases can be cancelled out.
- It is hard to definitively say that CJ is only measuring conceptual understanding and not procedural knowledge.
- Operationalizing conceptual and procedural knowledge may also come to question.


Michigan State University

A method used to measure conceptual understanding (usually in mathematics) that does not require traditional scoring rubrics. Comparative judgement is based off the psychological principle “that humans are better at comparing two objects against one another than they are at comparing one object against specified criteria.”

Comparative Judgement

JONES, I ... et al, 2013. Measuring conceptual understanding: the case of fractions. IN: Lindmeier, A.M. and Heinze, A. (eds). Proceedings of the 37th Conference of the International Group for the Psychology of Mathematics Education (PME 37), 28th July-2nd August 2013, Kiel, Germany, vol 3, pp. 113-120.

Measuring conceptual understanding: the case of fractions

Bisson, MJ., Gilmore, C., Inglis, M. et al. Measuring Conceptual Understanding Using Comparative Judgement. Int. J. Res. Undergrad. Math. Ed. 2, 141–164 (2016). https://doi.org/10.1007/s40753-016-0024-3

Measuring Conceptual Understanding Using Comparative Judgement

Students are asked an open-ended question relating to a topic of interest. Experts on this topic are then given two student responses, and they judge which of the two displays a better conceptual understanding of the concept without using a specified scoring rubric. This process is repeated several times, and a rank/scale is created for the responses from “worst” to “best.” Using statistical modeling, a z score parameter and standard error is created for the responses. Studies have found that comparative judgement displays internal consistency, reliability, and validity in varying (mathematical) contexts. 

Learn Before

Related