San Diego State University

This paper examines whether interaction data from Marmoset, an auto-grading system for programming assignments, can predict students' midterm and final exam grades. Features include per-task passing rate, testcase outcomes (public/release/hidden), time between submission and deadline, and number of submissions. The authors built decision-tree classification models (grouping students as good/satisfactory/poor performers) and linear-regression models (predicting numeric grades) from these features. The regression approach performed much better than classification, with a mean difference of 0.92 on a maximum grade of 120; when regression predictions were converted into performance categories for direct comparison with the classifiers, results also improved. The authors separately tested a regression model using only the submission-time-interval feature. Sections: Introduction, Related Work, Student Performance and Marmoset, Modeling Techniques, Features, Experiments and Evaluation.

Predicting student performance using data from an Auto-grading system

Usually in programming courses students would like to have multiple attempts to submit their coding assignments and get the feedback quickly in order to improve their code.  While this is desirable for the students, it would be very difficult for teachers and their assistants to review and grade each assignment multiple times. Therefore automating system is needed and Marmoset is a tool that allows users to submit codes several times. In this paper the authors want to assess if the data from Marmoset can be used for final and midterm exam grade predictions.

Introduction (Predicting student performance using data from an Auto-grading system)

Prior work on predicting student performance has targeted GPA, pass/fail rates, and exam grades using various data sources. Notably, one prior study used data from the PASTA auto-grading system to predict pass/fail rates in a final exam, but that dataset also included data from outside the auto-grading system. In contrast, the authors of this paper restrict their feature set exclusively to data from the Marmoset auto-grading system, without supplementing it with external student records.

Related Work (Predicting student performance using data from an Auto-grading system)

In the University of Vaterloo final and midterm exam contribute for 60%-90% of the final grade.  Marmoset is auto-grading system where students can submit coding assignments. Submitted code is tested on 3 different testcases: public, release and hidden. 

Student Performance and Marmoset (Predicting student performance using data from an 
Auto-grading system)

Reference for (Predicting student performance using data from an Auto-grading system)

The paper 'Predicting student performance using data from an Auto-grading system' evaluates two modeling techniques: decision trees and linear regression. Decision trees are built from the training set by choosing, at each split, the feature that yields the maximum possible information gain. For the regression approach, the authors use multivariate linear regression because the model includes several independent variables (features).

Modeling Techniques (Predicting student performance using data from an Auto-grading system)

The student-performance models use four categories of data from the Marmoset auto-grading system: (1) the per-task passing rate for each student's best submission, (2) public, release, and hidden test-case outcomes for the best submission, (3) the interval between submission time and the task deadline, and (4) the number of submissions. The dataset contains 28 tasks: the first 16 were due before the midterm exam, and the remaining 12 were due before the final exam.

Marmoset Auto-grading Features for Predicting Exam Performance

In the study of Marmoset auto-grading data, decision-tree classifiers predicted students as good, satisfactory, or poor performers, while linear-regression models predicted numeric exam grades. Classification models trained separately on the four feature categories performed poorly and assigned most students to the good-performance category; confusing poor performers with good performers was especially problematic. Regression performed substantially better, with a reported mean difference of 0.92 on a 120-point grading scale. Converting the predicted grades into performance categories also produced better results than direct classification. A separate regression model using only the interval between submission and deadline indicated a strong correlation between that feature and exam performance.

Learn Before

Related