Researchers conducted a proof-of-concept experiment in a simple setting to demonstrate the use of NLP to provide effective feedback. 949 participants from the UK were involved in this experiment and were paid $1 each. The learning activity consisted of identifying the Poison Ivy plant, which does not grow in the UK, and its characteristics. At the beginning of the activity, participants received one of two explanations, at random, about poison ivy. Each paragraph communicated one key feature of poison ivy: the first for leaves, the second for thorns, and colors. Providing these initial paragraphs induced "prior knowledge" for the rest of the experiment.
All participants then answered six multiple-choice questions in random order. Each question included a photo and contained three answer choices: "Poison ivy", "Not poison ivy", and "I don't know" (to discourage guessing). After the pre-test, participants answered four questions in a similar format as the pre-test questions but excluded the "I don't know" answer. Researchers also required participants to complete a short answer response with a minimum of 25 characters to explain their reasoning. After the exercises, participants were given one of two feedback paragraphs, each a paraphrase of one of the two initial explanations, and these were both uniformly randomized.
For the post-test portion of the experiment, after the participants read the feedback they were given, they were given a post-test with the same questions as the pre-test, but the question order was randomized. The feedback given to participants was chosen based on their responses, and they were given one of the two possible feedback paragraphs. Researchers measured the learning gain by subtracting the pre-test score from the post-test score.


Adaptive Natural-Language Targeting for Student Feedback- Experimental Procedure

The method used here targets feedback, not by semantic similarity, but by predicted results based on the end goal of knowledge gain. This requires the hand-creation of a set of feedback texts useful for the task at hand, and this reduces the labor cost relative to other hand-tailored methods.

University of Georgia

The goal of this study is to work toward developing more effective feedback-targeting systems for tutoring software, to improve user performance. In this study, researchers built a tutoring software system that uses Natural Language Processing (NLP) to provide adaptive feedback to students in an online learning task. 
- The researcher's system models the interaction between a student's natural language response and the software, which interprets students' responses and provides the best feedback action. This framework allows for the optimization of a reward signal. 
- Researchers showed that natural-language based targeting policies were able to choose optimal feedback from fewer exercises compared to multiple-choice based policies. The natural-language policies were effective even when tested on novel interactions. 

Adaptive Natural-Language Targeting for Student Feedback- Goal of Study

Kolchinski, Y. A., Ruan, S., Schwartz, D., & Brunskill, E. (2018). Adaptive natural-language targeting for student feedback. Proceedings of the Fifth Annual ACM Conference on Learning at Scale. doi:10.1145/3231644.3231684


Adaptive Natural-Language Targeting for Student Feedback

Natural language processing- The interaction between computers and humans via natural language
Reward signal- Pre-test to post-test gain, based on the scores
Semantic matching- A technique used in computer science to identify information that is semantically related.

Adaptive Natural-Language Targeting for Student Feedback- Relevant Terms

An important difference between human and software tutors is their ability to use natural language. Human tutors are effective in explaining concepts and gauging what students understand, but software is usually only effective in presenting natural-language material to students, and not in interpreting the natural-language responses of a student. It was found that using the natural language processing (NLP) targeted strategy was more effective in providing optimal feedback compared to traditional means of feedback targeting, like multiple-choice targeting. 

Adaptive Natural-Language Targeting for Student Feedback- Background

Previous tutoring systems have given feedback in several contexts, including correcting mistakes, providing hints, and metacognitive feedback meant to encourage good habits. The predominant ways in which previous systems targeted their feedback were based on task-specific inputs such as multiple-choice responses, computer code, and math equations written by the student. These systems are missing out because they use less expressive inputs for targeting feedback to students. Other systems have implemented the use of NLP for providing feedback, but these methods proved to have limitations of feedback targeting mechanisms, which depend on a combination of hand-tuned knowledge engineering and semantic matching. These methods are extremely effort-intensive and are not feasible on a larger scale.

Adaptive Natural-Language Targeting for Student Feedback- Previous Techniques

Adaptive Natural-Language Targeting for Student Feedback- Method of the Study

The researchers sought to evaluate other strategies of feedback-targeting, using the collected data, to evaluate the performance of the researcher's NLP strategy, which was used to pick out the right feedback message from the response. Researchers implemented and tested four strategies to evaluate in regards to learning gain: oracle, multiple-choice targeted policy, and two NLP targeted policies. There was a 95% confidence interval used in these tests. Participants' learning gains by four different one-questions policies were compared and results showed that policies targeted through NLP outperformed multiple-choice targeted strategies on all of the exercises except the last one (3/4 exercises). This shows the effectiveness of using NLP responses, rather than multiple-choice responses to provide the most effective feedback. Furthermore, compared to multiple-choice responses alone, the NLP strategy proved to contain more variation between questions. Unlike the multiple-choice models, the NLP models can generalize to unseen questions by learning a general notion of how students respond to prompts depending on what they know. These results show the promising effects of using natural-language inputs from students for feedback selection.


Learn Before

Related

Learn After