Knowledge Transfer by Discriminative Pre-training for Academic Performance Prediction

The needs for precisely estimating a student's academic performance have been emphasized with an increasing amount of attention paid to Intelligent Tutoring System (ITS). However, since labels for academic performance, such as test scores, are collected from outside of ITS, obtaining the labels is costly, leading to label-scarcity problem which brings challenge in taking machine learning approaches for academic performance prediction. To this end, inspired by the recent advancement of pre-training method in natural language processing community, we propose DPA, a transfer learning framework with Discriminative Pre-training tasks for Academic performance prediction. DPA pre-trains two models, a generator and a discriminator, and fine-tunes the discriminator on academic performance prediction. In DPA's pre-training phase, a sequence of interactions where some tokens are masked is provided to the generator which is trained to reconstruct the original sequence. Then, the discriminator takes an interaction sequence where the masked tokens are replaced by the generator's outputs, and is trained to predict the originalities of all tokens in the sequence. Compared to the previous state-of-the-art generative pre-training method, DPA is more sample efficient, leading to fast convergence to lower academic performance prediction error. We conduct extensive experimental studies on a real-world dataset obtained from a multi-platform ITS application and show that DPA outperforms the previous state-of-the-art generative pre-training method with a reduction of 4.05% in mean absolute error and more robust to increased label-scarcity.


INTRODUCTION
Predicting a student's future academic performance is a fundamental task for developing modern Intelligent Tutoring  Figure 1: Interactive features, such as student response and elapsed time for the response, are automatically recorded to the database whenever a student interacts with ITS. On the other hand, more complicated steps are necessary to obtain a test score: a student should take the test in the designated test center, receive the test score, and report the score to ITS. System (ITS) which aims to provide personalized learning experience by supporting educational needs of each individual. However, labels for academic performance, such as test scores, are often scarce since they are external to ITS. For example, as shown in Figure 1, test scores are not automatically collected inside of ITS. Obtaining a test score requires a student to take the test in the designated test center, receive the score, and report the score to ITS. Transfer learning is a commonly taken approach to address such label-scarcity problems across different domains of machine learning. In this framework, a model is first pre-trained to optimize auxiliary objectives with abundant data, and then fine-tuned on the task of interest. In Artificial Intelligence in Education (AIEd) community, [3] introduced Assessment Modeling (AM), a set of pre-training tasks for label-scarce educational problems including academic performance prediction. AM proposed a pre-training method where first, a masked interaction sequence is generated by replacing a set of interactive features which can serve as criteria for pedagogi-cal evaluation with artificial mask tokens. Then, given the masked interaction sequence, a model is pre-trained to predict the masked interactive features. The idea was borrowed from the Masked Language Modeling (MLM) pre-training method proposed in [7]. In the MLM pre-training method, given a masked word sequence where some words in the sequence are replaced with an artificial mask token, a model is pre-trained to predict the masked words. However, recently, [6] pointed out that the MLM pre-training method has poor sample efficiency and suffers from pre-train/fine-tune discrepancy due to the artificial mask token, and proposed a new discriminative pre-training method. Considering the problems are also inherent in AM, potential gains are expected to be obtainable when the discriminative pre-training method is applied to academic performance prediction.

. R e p o r t a s c o r e A u t o m a t i c a l l y c o l l e c t e d
To this end, we propose DPA, a transfer learning framework with Discriminative Pre-training tasks for Academic performance prediction. There are two models in DPA: a generator and a discriminator. In DPA's pre-training phase, the generator is trained to predict the masked interactive features in the same way as AM. Then, given a replaced interaction sequence which is generated by replacing the masked features with the generator's outputs, the discriminator is trained to predict whether each token in the sequence is the same as the one in the original interaction sequence. After the pre-training, the generator is thrown away and only the discriminator is fine-tuned on academic performance prediction. Also, we investigate diverse pre-training tasks for the generator and show that pre-training the generator to predict a student's response is more effective than to predict the correctness and timeliness of their response which were considered as the most pedagogical interactive features in AM. Extensive experimental studies conducted on a realworld dataset collected from a multi-platform ITS application show that DPA outperforms AM with a reduction of 4.05% in mean absolute error and more robust when the degree of label-scarcity increases. Through a series of ablation experiments, we show that DPA's sample efficient pretraining contributed most to the improvement from AM to DPA.

SANTA: A SELF-STUDY SOLUTION EQUIPPED WITH AN AI TUTOR FOR ENGLISH EDUCATION
In this paper, we conduct experiments on a real-world dataset obtained from Santa 1 , a multi-platform ITS with more than a million users in South Korea available through Android, iOS, and Web that exclusively focuses on the Test of English for International Communication (TOEIC) standardized examination. The publicly accessible version of the dataset was released under the name EdNet [4]. The TOEIC consists of two timed sections, Listening Comprehension (LC) and Reading Comprehension (RC). There are a total of 100 multiple choice exercises in each section, and the total score for each section is 495 in steps of 5 points. Santa provides learning experiences of solving exercises, studying explanations, and watching lectures. When a student consumes a specific learning content, Santa diagnoses their current academic status based on their learning activities records and recommends another learning content appropriate for their 1 https://aitutorsanta.com current position. Santa records diverse types of interactive features, such as student response, the duration of time the student took to respond, and the time interval between the current and previous learning activities. However, unlike the interactive features automatically collected from Santa, obtaining the official TOEIC score requires more steps: a student should register and pay for the test, take the test in the designated test center, receive the test score from the Educational Testing Service, and report the score to Santa (Figure 1). Santa collected students' TOEIC score data by offering small gifts to students when they report their scores.

TRANSFER LEARNING FOR ACADEMIC TEST PERFORMANCE PREDICTION
To overcome the label-scarcity problem in academic test performance prediction, we consider burgeoning machine learning discipline of transfer learning. There is an open issue of what information to transfer or which pre-training task is the most effective for academic test performance prediction. Previous studies proposed two types of pre-training methods for AIEd Tasks: interaction-based method which models students' dynamic learning behaviors [13,16,8,3], and content-based method which learns representations of learning contents [15,25,21,26,32]. [3] showed that interactionbased pre-training method outperforms content-based pretraining methods when the pre-trained model is fine-tuned on several label-scarce educational tasks including academic test performance prediction. Following this line of research, we propose a transfer learning framework where a model is pre-trained using only student interaction data, and finetune the pre-trained model on academic test performance prediction. In this paper, we consider the following interactive features: • eid : A unique ID assigned to an exercise solved by a student. There are a total of 14419 exercises in the dataset.
• part: Each exercise belongs to a specific part that represents the type of the exercise. There are a total of 7 parts in the TOEIC.
• response: Since the TOEIC consists of multiple choice exercises and there are four options for each exercise, a student response for a given exercise is one of the options, 'a', 'b', 'c', or 'd'.
• correctness: Whether a student responded correctly to a given exercise. Note that correctness is a coarse version of response since correctness is processed by comparing response with a correct answer for a given exercise.
• elapsed time: The amount of time a student spent on solving a given exercise.
• timeliness: Whether a student responded to a given exercise under the time limit. Note that timeliness is a coarse version of elapsed time since timeliness is processed by comparing elapsed time with the time limit recommended by domain experts for a given exercise.
• exp time: The amount of time a student spent on studying an explanation for an exercise they had solved.  Figure 2: The overall pre-training/fine-tuning process of DPA when each token in an interaction sequence is a set of eid, part, and response, and response is a feature being masked. mask and cls are special tokens for mask and classification, respectively, which are the same as the ones used in [7].
• inactive time: The time interval between the current and previous interactions.
In our experiments, we normalize the values of elapsed time, exp time, and inactive time so they are between 0 and 1 to stabilize the training process.  Figure 2, since the generator's outputs for the masked features are 'b' and 'a', a replaced interaction sequence is [(e419, part4, b), (e23, part3, b), (e4324, part3, a), (e5233, part1, a)]. Then, the discriminator takes the replaced interaction sequence I R as an input, and predicts whether each token in the sequence is the same as the one in the original interaction sequence (original) or not (replaced). After the pre-training, we throw away the generator and fine-tune the pre-trained discriminator on academic test performance prediction. We provide detailed explanations of each component in the generator and the discriminator, and training objective functions in the following subsections.

Interaction Embeddings
The embedding layer produces a sequence of interaction embedding vectors by mapping each interactive feature to an appropriate embedding vector. We take two different approaches to embed the interactive features depending on whether they are categorical (eid, part, response, correctness, and timeliness) or continuous (elapsed time, exp time, and inactive time) variables. If an interactive feature is a categorical variable, we assign unique latent vectors to possible values of the feature including special values for mask (mask ) and classification (cls). Take response as an example, there is an embedding matrix Eresponse ∈ R 6×d emb where each row vector is assigned to one of 'a', 'b', 'c', 'd', mask, and cls. If an interactive feature is a continuous variable, we assign a single latent vector to the feature. Then, an embedding vector for the feature is computed by multiplying the latent vector and a value of the feature. For instance, we compute an embedding vector for elapsed time as et * E elapsed time , where et is a specific value and E elapsed time ∈ R d emb is a latent vector assigned to elapsed time. Also, mask and classification for the continuous interactive features are indicated by setting their values to -1 and 0, respectively. Not only embeddings for interactive features, positional embeddings are also incorporated into Transformer-based models [29] to consider chronological order of each token. Rather than using conventional positional embeddings which stores an embedding vector for every possible position, we adopt axial positional embeddings [19] to further reduce memory usage. The final interaction embedding vector of dimension d emb for each time-step is the sum of all embedding vectors in the time-step. The interaction embedding layer is shared by both the generator and the discriminator.

Performer Encoder
Since its successful debut in Natural Language Processing (NLP) community, Transformer's attention mechanism has become a common recipe adopted across different domains of machine learning including speech processing [20], computer vision [1,9], and AIEd [23,2,11,24]. Compared to Recurrent Neural Network (RNN) family models, Transformer's attention mechanism has benefits of capturing longer-range dependencies and allowing parallel training, which enables the model to achieve better performance with less training time. However, despite these advantages, the time and memory complexities of computing the attention grow quadratically with respect to input sequence length, requiring demanding computing resources for training the model on long sequences. For instance, if L is input sequence length and d is dimension of query, key, and value vectors, Transformer's attention is computed as follows: where Q, K, V ∈ R L×d . The time and memory complexities for computing QK in the above equation are O(L 2 d) and O(L 2 ), respectively. Therefore, the cost for training Transformer becomes prohibitive with large L, preventing training the model even on a single GPU.
The problem of improving the efficiency of Transformer's attention mechanism is a common concern of machine learning community. Recent studies have proposed several methods to reduce the computing complexities lower than the quadratic degree with respect to input sequence length [19,30,17,27,5]. In this paper, we adopt Performer [5] since it uses reasonable memory and makes a better trade-off between speed and performance [28]. Performer approximates attention kernels through Fast Attention Via positive Orthogonal Random features (FAVOR+) approach. The R+part in FAVOR+ computes Q and K by applying random feature map φ : R d → R r + to each query vector q and key vector k in Q and K, respectively.
The FA-part leads to efficient attention mechanism computed as follows: where 1L ∈ R L is an all-ones vector of length L and diag is a diagonal matrix with the input as the diagonal. Unlike Transformer's attention mechanism which computes QK , applies the softmax, and multiplies the result and V , FA-VOR+ first computes K V , and multiplies the result and Q , reducing the quadratic time and memory complexities to linear ones with respect to input sequence length. Also, FA-VOR+ can model most attention kernels used in practice. If φ is a function given as below, FAVOR+ approximates the softmax attention kernel used in Transformer's attention mechanism.
Furthermore, the authors proposed a generalized attention kernel, an attempt to model a kernelizable attention mechanism beyond the commonly used softmax attention kernel, by defining φ as below.
where ReLU is the widely used Rectified Linear Unit function. In this paper, we use the generalized attention kernel instead of the softmax attention kernel since the former showed better performance in [5] and our preliminary experiments. Lastly, the O-part further reduces the variance of the estimator by making different random samples ws to be orthogonal to each other through the standard Gram-Schmidt orthogonalization process. For those who want to know more about theoretical details of FAVOR+, please refer [5].
With the efficient attention mechanism by FAVOR+, we propose the Performer encoder which is stacks of several identical reversible layers described in Figure 3. The reversible layer is based on Reversible Transformer [12,19] architecture to further improve memory efficiency in backpropagation. An input of the reversible layer x ∈ R L×d hidden is first chunked to x1, x2 ∈ R L×d hidden /2 . Then, scaled l2 normalization (ScaleNorm) [22] and FAVOR+-based multihead attention layer (MultiHeadAttn) are applied to x2, and the result is added to x1 to compute y1 ∈ R L×d hidden /2 . y1 = x1 + MultiHeadAttn(ScaleNorm(x2)).
After that, the scaled l2 normalization and point-wise feedforward layer (FeedForward) are applied to y1, and the result is added to x2, computing y2 ∈ R L×d hidden /2 . y2 = x2 + FeedForward(ScaleNorm(y1)). An output of the reversible layer y ∈ R L×d hidden is a concatenation of y1 and y2. We stack the reversible layer multiple times to allow the final model to sufficiently represent underlying data distribution.

Generator
The generator computes hidden representations [h G 1 , . . . , h G T ] by feeding the masked interaction sequence I M to a series of the interaction embedding layer (InterEmbedding), a point-wise feed-forward layer (GenFeedForward1), the Performer encoder (GenPerformerEncoder), and another pointwise feed-forward layer (GenFeedForward2): where I M E t , h G t ∈ R d emb and h GF t , h GP t ∈ R d gen hidden . Then, depending on whether the masked features are categorical or continuous variables, generator outputs are computed differently. If the masked features are categorical variables, the outputs are sampled from a probability distribution defined by the following softmax layer: If the masked features are continuous variables, the outputs are computed by the following sigmoid layer: . Similar to the case of categorical masked features, one can sample the outputs from a probability distribution defined by I M and parameters of the generator when the masked features are continuous variables. For instance, the outputs can be sampled from the Gaussian distribution where the mean and the variance are determined by I M and the generator's parameters. However, we make the outputs deterministic because sampling the outputs underperforms in our preliminary experiments when the masked features are continuous variables.

Discriminator
, and the sigmoid is applied to the last layer of the discriminator. After the pre-training, we slightly modify the discriminator by replacing the last layer with a layer having appropriate dimension for academic test performance prediction.

Training Objectives
The objective for pre-training is to minimize the following loss function: where GenLoss is the cross entropy (or mean squared error) loss function if the masked features are categorical (or continuous) variables, DisLoss is the binary cross entropy loss function, and 1 is the identity function. For ease of notation, we omit an index for each input sample in the above equation. If there are more than one masked features in each time-step (n > 1), the generator is trained under the multi-task leaning scheme. The objective for fine-tuning is to minimize the mean squared error loss between the model's predictions and score labels.

EXPERIMENTS 5.1 Dataset
The pre-training dataset consists of student interaction logs. The statistics of the dataset are summarized in Table 1. We exclude student interaction logs less than 15 in length to reduce noisy interactions from students using Santa just for trying out. Since most of the values of elapsed time, exp time, and inactive time are distributed in head areas as shown in Figure 4, we set their maximum to 300, 300, and 86400 seconds, respectively, and any values more than that are capped off to the maximum. We further normalize the values so that they are between 0 and 1 by dividing them by the maximum.
The fine-tuning dataset consists of test scores and student interaction logs before the test. Table 2 summarizes the statistics of the dataset. The number of score labels is far less than that of interactions in the pre-training dataset, which leads to the label-scarcity problem in academic test performance prediction. The minimum length of interactions before the test is 10 because Santa required students to solve at least 10 exercises before taking the test. The mean and median length of interactions before the test are longer than those in the pre-training dataset since students who decide to report their scores tend to be more serious about studying with Santa. As shown in Figure 4, most of the scores are distributed over the range from 700 to 900, and there are very few scores in the below 200 area. This has to do with the distribution of students using Santa whose initial and goal scores are usually in the range of 600 to 700 and higher than 800, respectively. Since the score labels are few in number, we perform 5-fold cross-validation by dividing the fine-tuning dataset into 5 splits and using 3/5, 1/5, and 1/5 of the dataset for training, validation, and test, respectively.

Training Details and Hyperparameters
We use the Mean Absolute Error (MAE) as the metric for academic test performance prediction. The list of hyperparameters and their values are described in Table 3. For each pre-training evaluation, the pre-trained model is fine-tuned and cross-validated on the validation set, which results in the same number of validation results as the number of pretraining evaluations. Then, we select the model with the best validation result and report an evaluation result of the model on the test set.

Effects of Generator's Pre-training Tasks
There are multiple interactive features to be masked in each token of the interaction sequence, which raises a question of how to construct a set of masked interactive features, and accordingly, which pre-training task for the generator is the most effective for academic test performance prediction. By default, all interactive features listed in Section 3 are taken as inputs for both the generator and discriminator. However, if response (or elapsed time) is masked, correctness (or timeliness) is excluded from the inputs and vice versa since there is an overlap of information that the features represent. For example, when both response and correctness are taken as inputs, and correctness is masked, the generator can predict the masked correctness by only looking at eid and response without considering other interactions, which leads to poor pre-training. The results are described in Table  4. The best result was obtained under the pre-training task of predicting response alone, which is slightly better than that of predicting correctness, and both response and correctness. Predicting correctness of student response is an important task in AIEd as can be seen from the large volume of studies about Knowledge Tracing. Also, [3] empirically showed that student response correctness is the most pedagogical interactive feature for academic test performance prediction. However, rather than pre-training a model to predict whether a student correctly responded to a given exercise, the pre-training task of predicting student response itself injects more fine-grained information into the model, which leads to the more effective pre-training for academic test performance prediction. Interestingly, the underperformed results were obtained when predicting elapsed time or timeliness in pre-training despite the benefits their information bring to several AIEd tasks [10,33,24]. We hypothesize that elapsed time and timeliness may introduce irrelevant noises and thus guide the model towards a direction inappropriate for academic test performance prediction. In the case of exp time and inactive time, we observed that the generator failed to learn to predict their values when only given the interactive features listed in Section 3, which leads to unstable pre-training. From these observations, in the following subsections, we conduct experimental studies based on the pre-training task of predicting response alone.

DPA vs. Baseline Methods
We compare DPA with the following pre-training methods: • No pre-training: We train the fine-tuning models only on the fine-tuning dataset.
• Autoencoding: Autoencoding (AE) is a generative pretraining method widely used across different domains of machine learning including AIEd [13,8]. Given an unmasked interaction sequence, AE pre-trains a model to reconstruct the input interaction sequence.
• Assessment Modeling: Assessment Modeling (AM) [3] is the previous state-of-the-art generative pre-training method for academic test performance prediction. In AM, a model takes a masked interaction sequence as an input and is pre-trained to predict masked features. AM is exactly the same as fine-tuning the pre-trained generator in DPA.
Also, we investigate whether DPA is effective with the following different fine-tuning models: • MLP: Multi-Layer Perceptron (MLP) is stacks of simple fully-connected layers. Given an interaction sequence, interaction embedding vectors of all time-steps are summed together to compute a fixed-dimensional vector which is fed to a series of the fully-connected layers.
• BiLSTM: Bi-directional Long Short-Term Memory (BiL-STM) is a model widely used for time series data prediction tasks. The global max pooling layer is applied on top of the BiLSTM layer to obtain a fixeddimensional intermediate representation from an input sequence of varying length.
• Transformer Encoder: Transformer Encoder is a series of several identical layers composed of a multi-head self-attention layer with the softmax attention kernel and a point-wise feed-forward layer. We set the Transformer encoder's attention window size to 512 due to the out of GPU memory occuring when training the Transformer encoder of 1024 attention window size on our single GPU machine.
As described in Table 5, transferring the pre-trained knowledge brings better results in most cases, and the best result is obtained from DPA. Especially, when the Performer encoder, the best performing fine-tuning model, is used as the fine-tuning model, DPA reduces MAE by 4.05%, 21.84%, and 38.05% compared to AM, AE, and No pre-training, respectively. Among the baseline pre-training methods excluding No pre-training, the worst result is obtained from AE beacuse the pre-training task of AE is much easier than  that of AM and DPA. We observed that the loss curve of AE converged to near zero within the first pre-training evaluation.

Robustness to Increased Label-scarcity
Since the motivation behind our proposal of DPA is the label-scarcity problem, we investigate how MAE changes at varying degrees of label-scarcity. Figure 5 and Table 6 describe the results when using 1/2, 1/4, and 1/8 of the total number of fine-tuning training samples. In all degrees of label-scarcity, DPA consistently outperforms AM. Also, DPA fine-tuned on 1/2, 1/4, and 1/8 of the dataset outperforms AM fine-tuned on the entire dataset, 1/2, and 1/4 of the dataset, respectively, which shows that DPA is more robust to label-scarcity than AM. Compared with No pre-training, the gap between No pre-training and the other two pre-training methods increases as the number of labels becomes scarce. Furthermore, the other two pre-training methods fine-tuned on 1/8 of the dataset outperform No pre-training fine-tuned on the entire dataset. Figure 4 described that score labels are mainly distributed over the specific ranges. We investigate how this biased distribution of score labels affects model predictions. The results are described in Figure 6. As expected, DPA severely underperforms when the score labels are below 200. Although these are natural results from a machine learning perspective, this is a serious problem from a perspective of educational service because students whose scores are lower than 200 are inaccurately diagnosed their academic status. It is also against the equity of education since not all students can receive the same level of educational service. There may be various research directions to solve this problem, such as generating pseudo labels, measuring prediction uncertainties, or even collecting more score labels. However, we don't go deeper into it any further and leave it as a future work.

Analysis of DPA
The previous experimental results showed that DPA makes better predictions and is more robust to label-scarcity than AM. However, where the gains from DPA are coming from is not obvious. We investigate what makes DPA outperform AM by comparing DPA and AM with the following set of ablation pre-training methods: • DPA60%: DPA60% is the same as DPA except the discriminator loss tokens only come from the masked interactions. Since we set the masked interaction ratio to 60%, DPA60% pre-trains the discriminator with 40% fewer loss tokens than DPA.
• RAM: Unlike DPA where there are the generator and the discriminator, in Replaced Assessment Modeling (RAM), there are two generators of different sizes, a small and large generator. The small generator is pretrained in the same way as AM. The large generator takes a replaced interaction sequence which is generated by replacing the masked features with the small generator's outputs as an input, and is pre-trained to predict the masked features. After the pre-training, we throw away the small generator and fine-tune the large generator. The sizes of the small and large generator are the same as those of DPA's generator and discriminator, respectively.  • AAM: All-tokens Assessment Modeling (AAM) is the same as RAM except the large generator is pre-trained to predict all features rather than just predicting the masked features. Figure 7 depicts graphical description of each pre-training method. The results are described in Table 7. In the following subsections, we analyze the results in aspects of pretrain/fine-tune discrepancy due to the mask token, discriminative vs. generative pre-training, and sample efficiency.

Pre-train/Fine-tune Discrepancy Due to Mask Token
During pre-training, the generator in AM sees the mask token which does not appear in fine-tuning, leading to pretrain/fine-tune discrepancy. [31,6] also raised the same issue found in the MLM pre-training method proposed in [7]. Since the discriminator in DPA does not see the mask token both in pre-training and fine-tuning, DPA does not suffer from the pre-train/fine-tune discrepancy, making it necessary to examine how much gain is obtainable from this issue. Comparing RAM and AM, removing the pre-train/fine-tune discrepancy slightly reduces MAE by 0.28%.

Discriminative vs. Generative Pre-training
The discriminator in DPA and the generator in AM are pretrained with different objectives. For each token to be pre-dicted, the discriminator pre-training loss comes from a discrimination error between the discriminator's output and an originality of the token. On the other hand, the generator is pre-trained by generation error between the generator's output and an identity of a token to be predicted. Comparing DPA with AAM, and DPA60 with RAM, the discriminative pre-training objective reduces MAE by 0.82% and 1.46%, respectively, over the generative pre-training objective.

Sample Efficiency
The loss tokens for pre-training the discriminator in DPA come from all interactions in the input interaction sequence. However, the generator in AM is pre-trained with loss tokens only coming from the masked interactions. Considering the masked interaction ratio is 60%, DPA is more sample efficient than AM since the discriminator is pre-trained with 40% more loss tokens than the generator. Comparing DPA with DPA60, and AAM with RAM, sample efficient pre-training brings reduction in MAE by 2.35% and 2.89%, respectively, showing that the sample efficiency contributed most to the improvement from AM to DPA.

CONCLUSION
In this paper, we proposed DPA, a transfer learning framework with discriminative pre-training tasks for academic performance prediction. Our experimental results showed the effectiveness of DPA for the label-scarce academic performance prediction task over the previous state-of-the-art generative pre-training method. Avenues of future research include investigating more effective pre-training tasks for academic performance prediction and pre-train/fine-tune relations in AIEd.