- One of the datasets comes from the Cognitive Tutor System called Bridge to Algebra and is from the 2006-2007 school year. This was one of the smaller, development datasets made public as part of the 2010 Knowledge Discover and Data mining competition. In this tutor, students answer algebra problems from their math curriculum which is split into sections. The problems consist of many steps that the students must answer to go to the next problem. A student no longer needs to answer steps of a given skill when the Cognitive Tutor’s Knowledge Tracing model believes the student knows the skill with probability 0.95 or greater. When a student has mastered all the skills in their current section they are allowed to move on to the next. The time for students using this system is determined by teachers. Twelve skills were chosen at random from this dataset for analysis (excluding skills such as “press enter” which do not represent math skills). There were an average of 122 students per skill in this dataset.
- Another dataset is collected from ASSISTments Platform’s Skill Builder problem sets. The ASSISTments Platform is an educational research platform better known for its elearning that provides web based math tutoring to 8th-10th grade students. Unlike the Cognitive Tutor System, students are forced to leave the tutor after 10 questions have been finished in one day and will come back to the tutor in a new day. If a student answers three questions correct in a row, they are “graduated” from the problem set. The help the tutorial provided is consist of a series of questions that break a problem into sub steps. A student can also request a hint, but requesting a hint will mark the student as getting the step wrong in the system. Only answers to the original questions are considered. The largest twelve Skill Builder datasets were selected from the ASSISTments Platform. There was an average of 1,200 students per problem set in this dataset. The highest student count problem sets were selected here because new day events are far more sparse in ASSISTments skill problem sets than the Cognitive Tutor skill problem sets.

The twelve datasets from each tutor were randomly divided into two equal parts by student, one part was used as the training set, the other as the testing set. 


Datasets for Prediction (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

Parameters were learned for each skill problem set individually. The parameters were unbounded and initial parameters were set to a Guess of 0.14, Slip of 0.09, Prior of 0.50 and Learn of 0.14, these initial values were the average parameter values across all skills in prior modeling work conducted on the ASSISTments tutor. For parameter learning, the new day observation (0 or 1) was presented as evidence in addition to the student responses. After training, the time and actual response values were given to the model as evidence for our new models to do the prediction one student at a time. In order to predict every response of each student in the test set, the student data for prediction was presented to the network in the following fashion: 
- for predicting the first question, no evidence was entered;
- for the second question, the new day information for that question and the actual response of first question were entered as evidence;
- for the third question, the first two new day information and responses information were entered as evidence.

Apply this procedure until the prediction of the last question. This predicting process is shown in Fig. 2. By applying this prediction process, the probability of student answering each question correctly was computed and saved. Fig. 2. The process of entering evidence data. Finally, we calculated the Residuals and AUC values between predictions and actual responses on same day events, new day events as well as overall events of the whole problem set. 

Prediction Procedure (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

- The prediction performance of the three models were calculated in terms of Residuals and AUC values between predictions and actual responses on same day events, new day events as well as overall events of the whole problem set. The model with higher AUC values for a problem set was deemed to be the more accurate predictor of that problem set. In addition, a two-tailed paired t-test was calculated between KT and KT-Forget and KT and KT-Slip.

- They first applied this to the datasets collected from Cognitive Tutor. Generally, the new KT-Forget model performed better on both the residuals and AUC compared to the regular KT model. Inversely, the KT-Slip model performed worse than expected. For the KT-Forget model, improved results were obtained both on residuals and AUC. Especially for the AUC, although KT-forget did not get significant improvement on new day events in terms of AUC (p value is 0.5175); however, it got significant improvement on same day events prediction and overall prediction (p value is 0.0178 and 0.0129), which means the performance of KT-Forget model is more accurate on predicting of Cognitive Tutor data compared to the regular KT model. Moreover, the better prediction performance also supported the hypothesis that students probably forget knowledge when it comes to a new day. For the KT-Slip model, the results of overall data’s AUC were worse but not significantly compared to regular KT. However, both same day and new day AUC were significantly worse, which overthrew our assumption that students may slip when it comes to a new day. Similarly, they applied their models to the ASSISTments datasets.

- It can be observed that the new models, both KT-Forget and KT-Slip lost to the regular KT model, especially on the AUC. They looked into the reason why the new models perform much worse and found that the way the data was collected lead to this result. As they mentioned in the previous section, students are forced to leave the tutor after a certain number of questions have been finished in one day and will come back to the tutor in a new day. Thus, we observed that the datasets collected from ASSISTments have much fewer new day events (average 1 per student) and is not as amenable to a time analysis as the Cognitive Tutor data which has many new days per student and students experience the new day more naturally. Therefore, the results obtained from Cognitive Tutor are more practical for this analysis.

Prediction Result Analysis (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

To evaluate the performance of the KT-Forget and the KT-Slip models, the researchers used a Cognitive Tutor dataset and ASSISTments dataset to test the real world utility of these models by comparing their predictive performance with a standard KT model. For each problem set, which represents a certain skill, they trained regular KT, KT-Forget and KTSlip models to make predictions on all the question responses of each student. Then the Residuals and AUC is calculated for predictions and actual responses on same day events, new day events as well as overall events to analyze the three models’ performance. 

University of Michigan - Ann Arbor

- Introduction
- Time Model Design
- Model Performance Evaluations
- Contributions
- Discussions and Future Work

Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing

http://web.cs.wpi.edu/~nth/pubs_and_grants/papers/2011/EDM%202011/Qiu%20Does%20Time%20Matter.pdf 

Reference for Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing

The knowledge tracing model has been widely used to model student knowledge and learning over time. It assumes that each skill has two knowledge parameters, prior and learn; and two performance parameters, slip and guess. The learn parameter represents the probability that a student will transition between the unlearned and the learned state after each question. The slip parameter is the probability that a student who understands a skill can make a careless mistake and the guess parameter is the probability a student may answer correctly in spite of not knowing the skill. When using the standard Knowledge Tracing (KT) model, it is assumed that the students’ probability of making the transition from the unlearned to the learned state is constant opportunities (or questions). Many researchers assume that student performance a minute later is the same as the next day. In the real world, coming into class on a new day may result in a student forgetting the material or a higher probability of them slipping. By taking this real world fact into consideration, this paper looks into how KT performs on each new day’s responses. A new day’s response is defined as a response that occurred on a later calendar date than the student’s previous response to a question of the same skill. KT’s new day error is far higher than same day error. A residual analysis showed that KT was largely over predicting student performance on each new day response. Based on the residual result, two hypotheses explain this phenomenon: 
- 1) that students may forget between days and
- 2) that students may slip when answering the first question on a new day.

The slip hypothesis only affects the model’s prediction of new day events while the forget hypothesis could affect prediction of subsequent responses since it hypothesizes a change in the latent of knowledge. Two new models were developed based on Knowledge Tracing: a KT-Forget Model and a KT-Slip Model, where a new day variable is taken into account to affect either students’ knowledge or performance. To implement this, a new split-parameter KT model was introduced, which allowed learning a different forget parameter for new day opportunities than for same day but learn only a single learn rate parameter for each.

Introduction (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

When using the Knowledge Tracing model, it is assumed that the student’s probability of making the transition from the unlearned to the learned state is not changing across opportunities, while in the real world students may forget the previously learned knowledge when coming into class on a new day. This fact assumes that there is a great possibility that a student’s forgetting rate is not zero. Poor performance on a new day may also suggest that students may not actually be “forgetting” but instead, they might just be “slipping.” The researchers used Bayesian networks and Expectation Maximization to detect whether time had any influence on the forget parameter and the slip parameter of the KT model. The model with the better predictive accuracy will indicate the better cognitive explanation of the data. 

Time Model Design (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

Model Performance Evaluations (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

This paper makes two contributions. First, they showed assumptions made in Knowledge Tracing model, that students don’t forget, is false. While this might not be terribly surprising, they identify a particular situation in which the standard KT model has systematic errors in predicting student performance, which is on new day responses. Secondly, they present a model to account for this phenomenon which does a reliably better job of fitting student data in some datasets. This is significant as KT has proved itself to be a very effective model, difficult to improve upon. It is also noteworthy that KT is easily interpretable, and it is beneficial to be able to have a clean model that fits easily into the Bayesian framework and inherits this interpretability. Their contribution is that researchers should pay attention to “time” and we have demonstrated a method that takes this into account and improves modeling performance. 

Contributions (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

In this work they attempt to model the time factor to better predict students’ learning performance in intelligent tutoring systems. Due to their experiment results that new KTforget model worked very well on Cognitive Tutor datasets while failed on ASSISTments Tutor datasets, they need to further investigate into the real reasons which caused this. Thus, they would like to know when using KT-forget model is not beneficial. In this paper they only made two assumptions that the parameters “forget” and “slip” will be affected by time factor. They have not yet looked into the performance of other parameters that might be affected by time, for example: students may have a fresh mind and learn more on a new day, which means a new parameter “learn new day” should be modeled. Also, it is possible that “time” should connect to these two parameters at once. They will keep on delving into these possibilities to see whether further improvement incorporating time can be obtained. If this is achieved in the future, They can build an ensemble model that combines regular KT’s results on the same day with the new model’s results on a new day. Their work only focuses on whether students answer the questions in one day or in a new day, they do not pay attention to the intervals between the same day and a new day. Pavlik and Anderson’s study showed that longer intervals should have a greater impact more on students’ performance while shorter intervals may have very little effect on actual responses. These topics deserve further investigation to figure out how to leverage the valuable time information and build better user models.

Learn Before

Related

Learn After