The knowledge tracing model has been widely used to model student knowledge and learning over time. It assumes that each skill has two knowledge parameters, prior and learn; and two performance parameters, slip and guess. The learn parameter represents the probability that a student will transition between the unlearned and the learned state after each question. The slip parameter is the probability that a student who understands a skill can make a careless mistake and the guess parameter is the probability a student may answer correctly in spite of not knowing the skill. When using the standard Knowledge Tracing (KT) model, it is assumed that the students’ probability of making the transition from the unlearned to the learned state is constant opportunities (or questions). Many researchers assume that student performance a minute later is the same as the next day. In the real world, coming into class on a new day may result in a student forgetting the material or a higher probability of them slipping. By taking this real world fact into consideration, this paper looks into how KT performs on each new day’s responses. A new day’s response is defined as a response that occurred on a later calendar date than the student’s previous response to a question of the same skill. KT’s new day error is far higher than same day error. A residual analysis showed that KT was largely over predicting student performance on each new day response. Based on the residual result, two hypotheses explain this phenomenon: 
- 1) that students may forget between days and
- 2) that students may slip when answering the first question on a new day.

The slip hypothesis only affects the model’s prediction of new day events while the forget hypothesis could affect prediction of subsequent responses since it hypothesizes a change in the latent of knowledge. Two new models were developed based on Knowledge Tracing: a KT-Forget Model and a KT-Slip Model, where a new day variable is taken into account to affect either students’ knowledge or performance. To implement this, a new split-parameter KT model was introduced, which allowed learning a different forget parameter for new day opportunities than for same day but learn only a single learn rate parameter for each.

Introduction (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

When using the Knowledge Tracing model, it is assumed that the student’s probability of making the transition from the unlearned to the learned state is not changing across opportunities, while in the real world students may forget the previously learned knowledge when coming into class on a new day. This fact assumes that there is a great possibility that a student’s forgetting rate is not zero. Poor performance on a new day may also suggest that students may not actually be “forgetting” but instead, they might just be “slipping.” The researchers used Bayesian networks and Expectation Maximization to detect whether time had any influence on the forget parameter and the slip parameter of the KT model. The model with the better predictive accuracy will indicate the better cognitive explanation of the data. 

Time Model Design (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

To evaluate the performance of the KT-Forget and the KT-Slip models, the researchers used a Cognitive Tutor dataset and ASSISTments dataset to test the real world utility of these models by comparing their predictive performance with a standard KT model. For each problem set, which represents a certain skill, they trained regular KT, KT-Forget and KTSlip models to make predictions on all the question responses of each student. Then the Residuals and AUC is calculated for predictions and actual responses on same day events, new day events as well as overall events to analyze the three models’ performance. 

Model Performance Evaluations (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

This paper makes two contributions. First, they showed assumptions made in Knowledge Tracing model, that students don’t forget, is false. While this might not be terribly surprising, they identify a particular situation in which the standard KT model has systematic errors in predicting student performance, which is on new day responses. Secondly, they present a model to account for this phenomenon which does a reliably better job of fitting student data in some datasets. This is significant as KT has proved itself to be a very effective model, difficult to improve upon. It is also noteworthy that KT is easily interpretable, and it is beneficial to be able to have a clean model that fits easily into the Bayesian framework and inherits this interpretability. Their contribution is that researchers should pay attention to “time” and we have demonstrated a method that takes this into account and improves modeling performance. 

Contributions (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

In this work they attempt to model the time factor to better predict students’ learning performance in intelligent tutoring systems. Due to their experiment results that new KTforget model worked very well on Cognitive Tutor datasets while failed on ASSISTments Tutor datasets, they need to further investigate into the real reasons which caused this. Thus, they would like to know when using KT-forget model is not beneficial. In this paper they only made two assumptions that the parameters “forget” and “slip” will be affected by time factor. They have not yet looked into the performance of other parameters that might be affected by time, for example: students may have a fresh mind and learn more on a new day, which means a new parameter “learn new day” should be modeled. Also, it is possible that “time” should connect to these two parameters at once. They will keep on delving into these possibilities to see whether further improvement incorporating time can be obtained. If this is achieved in the future, They can build an ensemble model that combines regular KT’s results on the same day with the new model’s results on a new day. Their work only focuses on whether students answer the questions in one day or in a new day, they do not pay attention to the intervals between the same day and a new day. Pavlik and Anderson’s study showed that longer intervals should have a greater impact more on students’ performance while shorter intervals may have very little effect on actual responses. These topics deserve further investigation to figure out how to leverage the valuable time information and build better user models.

Discussions and Future Work (Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing)

- Introduction
- Time Model Design
- Model Performance Evaluations
- Contributions
- Discussions and Future Work

University of Michigan - Ann Arbor

Research addressing the prediction of a user's return time to a learning tool, based on their previous interactions, includes studies such as "Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment" and "Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing".

Research Papers on Predicting User Return Time Based on Previous Interactions

http://web.cs.wpi.edu/~nth/pubs_and_grants/papers/2011/EDM%202011/Qiu%20Does%20Time%20Matter.pdf 

Reference for Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing

Reference for Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment

Does Time Matter? Modeling the Effect of Time in Bayesian Knowledge Tracing

The paper "Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment" proposes the Deep Attentive Study (DAS) model to predict the probability of student dropout from an ongoing study session. Specifically tailored for mobile learning environments, the study utilizes the Santa dataset (containing student interactions with an AI tutor) to evaluate the DAS model against baseline models such as LSTM and GRU networks.

Learn Before

Related

Learn After