Adjusted Rand Index (ARI) score was the highest when the Euclidean distance and average cluster linkage was used. (Synthetic-5 dataset).
Then the authors used this optimal parameters to test its performance on ASSISTments 2009-2010 dataset. They used different number of clusters and measured the accuracy using AUC score. 

San Diego State University

The Results and Analysis section of the Knowledge Query Network (KQN) paper evaluates the model along four dimensions: (1) Correctness Prediction, (2) Knowledge Interaction Visualization, (3) Skill Domain Analysis, and (4) Sensitivity Analysis of the Vector Space Dimensionality.

Results and Analysis (Knowledge Query Network for Knowledge Tracing)

They used Area Under the ROC curve to measure the models performances. The authors compared KQN and DKT + KQN to BKT (Bayesian Knowledge Tracing), IRT(Item Response Theory), DKT (Deep Knowledge Tracing), DKVMN (Deep Key Value Memory Network) models.  Overall, KQN showed the best performance:



Correctness Prediction (Knowledge Query Network for Knowledge Tracing)

The authors visualized correctness of the predictions with respect to particular skills for one student  using the ASSISTments 2009-2011 dataset. Even though after visualizations it was found that the results were intuitively correct like the value after correctly solving a problem with a particular skill increased the value of correctly solving another problem that included that skill, there were cases when after incorrectly solving a problem the value of correctly solving the problem which included that skill still increased. This is the same problem that was in case of DKT models which needs to be addressed. 

Knowledge Interaction Visualization (Knowledge Query Network for Knowledge Tracing)

Skill Domain Analysis (Knowledge Query Network for Knowledge Tracing)

In order to measure the impact of the embedding dimension on the models performance different values were tried, d = $d_{opt}$, $0.5d_{opt}$, 2$d_{opt}$.
For each pair from these three pairwise skill distance was calculated as:

$$\xi_{d_1, d_2}  = \frac {\sum_{i \gt j}|pdist_{d1}  (s_i, s_j) - pdist_{d_2}(s_i, s_j)|} {\begin{pmatrix} N\\ 2 \end {pmatrix}}$$
$$d_1 \neq d_2$$

$pdist_d(s_i, s_j)$ refers to the pairwise distance and N is number of skills.
Then the $\xi$ is compared to average pairwie distance $\mu$:

$$\mu_d =   \frac {\sum_{i \gt j}pdist_{d}  (s_i, s_j) } {\begin{pmatrix} N\\ 2 \end {pmatrix}}$$
According to this experiment, KQN learned the skill relationship in a better manner when d was higher. 
Additionally, the authors used Mantel tests to measure the similarity between 2 distance matrices. It showed that when predicting the probability of the correctness as well as for learning relation between skill vectors KQN model is stable for different d values.

Learn Before

Related