University of Georgia

Researchers conducted a proof-of-concept experiment in a simple setting to demonstrate the use of NLP to provide effective feedback. 949 participants from the UK were involved in this experiment and were paid $1 each. The learning activity consisted of identifying the Poison Ivy plant, which does not grow in the UK, and its characteristics. At the beginning of the activity, participants received one of two explanations, at random, about poison ivy. Each paragraph communicated one key feature of poison ivy: the first for leaves, the second for thorns, and colors. Providing these initial paragraphs induced "prior knowledge" for the rest of the experiment.
All participants then answered six multiple-choice questions in random order. Each question included a photo and contained three answer choices: "Poison ivy", "Not poison ivy", and "I don't know" (to discourage guessing). After the pre-test, participants answered four questions in a similar format as the pre-test questions but excluded the "I don't know" answer. Researchers also required participants to complete a short answer response with a minimum of 25 characters to explain their reasoning. After the exercises, participants were given one of two feedback paragraphs, each a paraphrase of one of the two initial explanations, and these were both uniformly randomized.
For the post-test portion of the experiment, after the participants read the feedback they were given, they were given a post-test with the same questions as the pre-test, but the question order was randomized. The feedback given to participants was chosen based on their responses, and they were given one of the two possible feedback paragraphs. Researchers measured the learning gain by subtracting the pre-test score from the post-test score.


Adaptive Natural-Language Targeting for Student Feedback- Experimental Procedure

Kolchinski, Y. A., Ruan, S., Schwartz, D., & Brunskill, E. (2018). Adaptive natural-language targeting for student feedback. Proceedings of the Fifth Annual ACM Conference on Learning at Scale. doi:10.1145/3231644.3231684


Learn Before

Related