Weighted Bloom’s Index.

• Bloom’s taxonomy of learning identifies six levels of understanding on any topic and they are used to create a Weighted Bloom’s Index, which summarizes the average Bloom’s level of exam questions weighted by the points possible. Level 1 and 2 questions test lower-order cognitive skills. Level 3–6 questions assess higher-order cognitive skills. Specific Weighted Bloom’s Index values conform to each Bloom’s level
• A Weighted Bloom’s Index for every exam in the study is calculated by recruiting three experienced TAs to assign a Bloom’s level to every exam question given in each quarter. If two of the three raters agree, the experimenters assigned the majority rule rating.When ratings were sequential, the middle value is accepted. When ratings were nonsequential, the arithmetic average is accepted
• The raters were trained in “Blooming” exams by one of the authors (M.P.W.) and assessed questions presented in a common format and random order.
• The raters knew that they were assessing Biology 180 exam questions. But they were blind to the quarter and year, the study’s intent, and to the hypothesis being tested by the ratings. Point values for each question were omitted to avoid any bias introduced by high- vs. low-point-value questions

Predicted Exam Score

• Predicted Exam Score (PES) is calculated by recruiting three experienced Tas (different from the individuals who did the Bloom’s ratings) who were also experienced graders to predict the average number of points that students would receive on each part of each exam question in the study.
• The PES raters were blind to the study’s intent and the hypothesis being tested with the predicted-points data.
• After a norming session when they discuss their predictions for average points, each grader worked individually and not discussed. The arithmetic average of the three ratings submitted are computed to arrive at a single predicted-average-points value for each of the exam-question parts after the norming session.