pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models
Knowledge tracing (KT) is the task of using students' historical learning
interaction data to model their knowledge mastery over time so as to make
predictions on their future interaction performance. Recently, remarkable
progress has been made of using various deep learning techniques to solve the
KT problem. However, the success behind deep learning based knowledge tracing
(DLKT) approaches is still left somewhat mysterious and proper measurement and
analysis of these DLKT approaches remain a challenge. First, data preprocessing
procedures in existing works are often private and/or custom, which limits
experimental standardization. Furthermore, existing DLKT studies often differ
in terms of the evaluation protocol and are far away real-world educational
contexts. To address these problems, we introduce a comprehensive python based
benchmark platform, \textsc{pyKT}, to guarantee valid comparisons across DLKT
methods via thorough evaluations. The \textsc{pyKT} library consists of a
standardized set of integrated data preprocessing procedures on 7 popular
datasets across different domains, and 10 frequently compared DLKT model
implementations for transparent experiments. Results from our fine-grained and
rigorous empirical KT studies yield a set of observations and suggestions for
effective DLKT, e.g., wrong evaluation setting may cause label leakage that
generally leads to performance inflation; and the improvement of many DLKT
approaches is minimal compared to the very first DLKT model proposed by Piech
et al. \cite{piech2015deep}. We have open sourced \textsc{pyKT} and our
experimental results at \url{https://pykt.org/}. We welcome contributions from
other research groups and practitioners.