ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation

Session-based recommendation has received growing attention recently due to the increasing privacy concern. Despite the recent success of neural session-based recommenders, they are typically developed in an offline manner using a static dataset. However, recommendation requires continual adaptation to take into account new and obsolete items and users, and requires"continual learning"in real-life applications. In this case, the recommender is updated continually and periodically with new data that arrives in each update cycle, and the updated model needs to provide recommendations for user activities before the next model update. A major challenge for continual learning with neural models is catastrophic forgetting, in which a continually trained model forgets user preference patterns it has learned before. To deal with this challenge, we propose a method called Adaptively Distilled Exemplar Replay (ADER) by periodically replaying previous training samples (i.e., exemplars) to the current model with an adaptive distillation loss. Experiments are conducted based on the state-of-the-art SASRec model using two widely used datasets to benchmark ADER with several well-known continual learning techniques. We empirically demonstrate that ADER consistently outperforms other baselines, and it even outperforms the method using all historical data at every update cycle. This result reveals that ADER is a promising solution to mitigate the catastrophic forgetting issue towards building more realistic and scalable session-based recommenders.

The major challenge of continual learning is catastrophic forgetting [6,23]. That is, a neural model updated on new data distributions tends to forget old distributions it has learned before. A naive solution is to retrain the model using all historical data every time. However, it suffers from severe computation and storage overhead in large-scale recommendation applications.
To this end, we propose to store a small set of representative sequences from previous data, namely exemplars, and replay them each time when the recommendation model needs to be trained on new data. Methods using exemplars have shown great success in different continual learning [3,29] and reinforcement learning [2,32] tasks. In this paper, we propose to select representative exemplars of an item using an herding technique [29,36], and its exemplar size is proportional to the item frequency in the near past. To enforce a stronger constraint on not forgetting previous user preferences, we propose a regularization method based on the well-known knowledge distillation technique [12]. We propose to apply a distillation loss on the selected exemplars to preserve the model's knowledge. The distillation loss is further adaptively interpolated with the regular cross-entropy loss on the new data by considering the difference between new data and old ones to flexibly deal with different new data distributions.
Altogether, (1) we are the first to study the practical continual learning setting for the session-based recommendation task; (2) we propose a method called Adaptively Distilled Exemplar Replay (ADER) for this task, and benchmark it with state-of-the-art continual learning techniques; (3) experiment results on two widely used datasets empirically demonstrate the superior performance of ADER and its ability to mitigate catastrophic forgetting. 1

Session-based Recommendation
Session-based recommendation (SR) can be formulated as a sequence learning problem to be solved by recurrent neural networks (RNNs). The first work (GRU4Rec, [11]) uses a gated recurrent unit (GRU) to learn session representations from previous clicks. Based on GRU4Rec, [10] proposes new ranking losses on relevant sessions, and [34] proposes to augment training data. Attention operation is first used by NARM [18] to pay attention to specific parts of the sequence.
Base on NARM, [20] proposes STAMP to model users' general and short-term interests using two separate attention operations, and [30] proposes RepeatNet to predict repetitive actions in a session. Motivated by the recent success of Tansformer [35] and BERT [5] for language model tasks, [16] proposed SASRec using Transformer, and [33] proposed BERT4Rec to model bi-directional information. Despite the broad exploration and success, the above methods are all studied in a static and offline manner. Recently, the incremental and steaming nature of SR is pointed out by [9,27].
Besides neural approaches, several non-parametric methods have been proposed. [15] proposed SKNN to compare the current session with historical sessions in the training data. Lately, variations [8,21] of SKNN have been proposed to consider the position of items in a session or the timestamp of a past session. [7,[24][25][26] applies a non-parametric structure called context tree. Although these methods can be efficiently updated, the realistic continual learning setting and the corresponding forgetting issue remain to be explored.

Continual Learning
The major challenge for continual learning is catastrophic forgetting [6,23]. Methods designed to mitigate catastrophic forgetting fall into three categories: regularization [17,19,38], Exemplar Replay [3,4,29] and dynamic architectures [22,31]. Methods using dynamic architectures increase model parameters throughout the training process, which leads to an unfair comparison with other methods. In this work, we focus on the first two categories. Regularization methods add specific regularization terms to consolidate knowledge learned before. [19] introduces knowledge distillation [12] to penalize model logit change, and it is widely employed by [3,14,29,37,39]. [1,17,38] propose to penalize changes on parameters that are crucial to old knowledge according to various importance measures.

METHODOLOGY
In this section, we first introduce some background in Section 3.1 and a formulation of the continual learning setup in Section 3.2. In Section 3.3, we propose our method called "Adaptively Distilled Exemplar Replay" (ADER).

Background on Neural Session-based Recommenders
A user action in SR is a click or view on an item, and the task is to predict the next user action based on a sequence of user actions in the current web-browser session. Existing neural models f (θ ) typically contain two modules: an feature extractor ϕ(x) to compute a compact sequence representation of the sequence x of previous user actions, and an output layer ω(ϕ(x)) to predict the next user action. Various recurrent neural networks [10,11] and attention mechanisms [16,18,20] have been proposed for ϕ, and the common choices for the output layer ω is fully-connect layers [11] or bi-linear decoders [16,18]. In this paper, we base our comparison on SASRec [16], and we refer readers to model details in the original paper to avoid verbosity. Nevertheless, the techniques proposed and compared in this paper are agnostic to f (θ ), therefore, a more thorough comparison using different f (θ ) are left for interesting future work.

Formulation of Continual Learning for Session-based Recommendation
In this section, we formulate the continual learning setting for the session-based recommendation task to simulate the realistic use cases of training a recommendation model continually. To be specific, at an update cycle t, the recommendation model f (θ t −1 ) obtained until the last update cycle t − 1 needs to be updated with new incoming data the incoming data D t +1 before the next update cycle t + 1. A visualization of the continual learning setup is illustrated in Fig. 1, where a recommendation model is continually trained and tested upon receiving data in sequential update cycles.

Proposed Solution: Adaptively Distilled Exemplar Replay (ADER)
3.3.1 Exemplar Replay. To alleviate the widely-recognized catastrophic forgetting issue in continual learning, the model needs to preserve old knowledge it has learned before. To this end, we propose to store past samples,a.k.a Algorithm 1 ADER: ExemplarSelection at cycle t Train θ t with loss in Eq. (4) end while Compute E t using Algorithm 1 with θ t and M t computed by Eq. (1) Output: updated θ t and new exemplar set E t exemplars, and replay them periodically to preserve previous knowledge. To maintain a manageable memory footprint, we only store a fixed total number of exemplars throughout the entire continual learning process. Two decisions need to be made at each cycle t: (1). how many exemplars should be stored for each item/label? (2). what is the criterion for selecting exemplars of an item/label? First, we design the number of exemplars of each appeared item in I t (i.e. the set of appeared items until cycle t) to be proportional to its appearance frequency. In other words, more frequent and popular items contribute a larger portion of selected exemplars to be replayed to the next cycle. Suppose we store N exemplars in total, the number of exemplars m t,i at cycle t for a item i ∈ I t is: where the second term is the probability that item i appears in the current update cycle, as well as in the exemplars E t −1 we kept from the last cycle. Therefore, the exemplar sizes of different items to be select in the cycle t can be encoded as Second, we need to decide which samples to select as exemplars for each item. We propose to use a herding technique [29,36] to select the most representative sequences of an item in an iterative manner based on the distance to the mean feature vector of the item. In each iteration, one sample from D t ∪ E t −1 that best approximates the average feature vector (µ) over all training examples of this item (y) is selected to E t . The details are presented in Algorithm 1.

Adaptive Distillation on Exemplars.
The number of exemplars should be reasonably small to reduce memory overhead. As a consequence, the constraint to prevent the recommender forgetting previous user preference patterns is not strong enough. To enforce a stronger constrain on not forgetting old user preference patterns, we propose to use a knowledge distillation loss [12] on exemplars to better consolidate old knowledge At a cycle t, the set of exemplars to be replayed is E t −1 and the set of items till the last cycle is I t −1 , the proposed knowledge distillation (KD) loss is written as: where [p 1 , . . . ,p |I t −1 | ] is predicted distribution (softmax of logits) over I t −1 generated by f (θ t −1 ), and [p 1 , . . . , is the prediction of f (θ t ) over I t −1 . L K D measures the difference between the outputs of the previous model and the current model on exemplars, and the idea is to penalize prediction changes on items in previous update cycles. L K D defined above is interpolated with a regular cross-entropy (CE) loss computed w.r.t. D t defined below: In practice, the size of incoming data and the number of new items varies in different cycles, therefore, the degree of need to preserve old knowledge varies. To this end, we propose an adaptive weight λ t to combine L K D with L C E : In general, λ t increases when the ratio of the number of old items to that of new items increases, and when the ratio of the exemplar size to the current data size increases. The idea is to rely more on L K D when the new cycle contains fewer new items or fewer data to be learned. The overall training procedure of ADER is summarized in Algorithm 2.
As in [11,16,18,20], we remove sessions of length 1 and items that appear less than 5 times. To simulate the continual learning scenario, we split the model update cycle of DIGINETICA by weeks and YOOCHOOSE by days as its volume is much larger. Different time spans also resemble model update cycles at different granulates. In total, 16 update cycles are used to continually train the recommender on both datasets. 10% of the training data of each update cycle is randomly selected as a validation set. Statistics of split datasets are summarized in Table 1. We can see that YOOCHOOSE is less dynamic, indicated by the tiny fraction of actions on new items, that is, old items heavily reappear.

Evaluation Metrics
Two commonly used evaluation metrics are used: (1)

Baseline Methods
Several widely adopted baselines in continual learning literature are compared: • Finetune: At each cycle, the recommender trained till the last task is trained with the data from the current task.
• Dropout [28]: Dropout [13] is recently found by [28] that it effectively alleviates catastrophic forgetting. Based on Finetune, we applied dropout to every self-attention and feed-forward layer. • EWC [17]: It is a well-known method to alleviate forgetting by regularizing parameters important to previous data estimated by the diagonal of a Fisher information matrix computed w.r.t. exemplars.
• ADER (c.f. Algorithm 2): The proposed method using adaptively distilled exemplars in each cycle with dropout.
• Joint: In each cycle, the recommender is trained (with dropout) using data from the current and all historical cycles. This is a common performance "upper bound" for continual learning.
The above methods are applied on top of the state-of-the-art base SR recommender SASRec [16] using 150 hidden units and 2 stacked self-attention blocks. During continual training, we set the batch size to be 256 on DIGINETICA and 512 on YOOCHOOSE. We use Adam optimizer with a learning rate of 5e-4. A total of 100 epochs are trained, and early stop is applied if validation performance (Recall@20) does not improve for 5 consecutive epochs. Other hyper-parameters are tuned to maximize Recall@20. The dropout rate of Dropout, ADER, and Jointis set to 0.3; 30,000 exemplars are used by default for EWC and ADER; λ base of ADER is set to 0.8 on DIGINETICA and 1.0 on YOOCHOOSE.

Overall Results on Two Datasets
Results averaged over 16 update cycles are presented in Table 2, and several interesting observations can be noted: • Finetune already works reasonably well, especially on the less dynamic YOOCHOOSE dataset. The performance gap between Finetune and Joint is less significant than typical continual learning setups [14,19,29,37]. The reason is that catastrophic forgetting is not severe since old items can frequently reappear in recommendation tasks.
• EWC only outperforms Finetune marginally, and it performs worse than Dropout.
• Dropout is effective, and it notably outperforms Finetune, especially on the more dynamic DIGINETICA dataset.
• ADER significantly outperforms other methods, and the improvement margin over other methods is larger on the more dynamic DIGINETICA dataset. Furthermore, it even outperforms Joint. This result empirically reveals that ADER is a promising solution for the continual recommendation setting by effectively preserving user preference patterns learned before.
Detailed disentangled performance at each update cycle is plotted in Figure 2. We can see that the advantage of ADER is significant on the more dynamic DIGINETICA dataset. On the less dynamic YOOCHOOSE dataset, the gain of ADER mainly comes from the more dynamic starting cycles with relatively more actions on new items. At later stable cycles with few new items, different methods show comparable performance, including the vanilla Finetune.

In-depth Analysis
In following experiments, we conducted an in-depth analysis of the results on the more dynamic DIGINETICA dataset.  4 5 6 7 8 9 10 11 12 13 14 15  using 30k exemplars, we tested using only 10k/20k exemplars, and results are shown in Table 3. We can see that the performance of ADER only drops marginally as exemplar size decreases from 30k to 10k. This result reveals that ADER is insensitive to the number of exemplars, and it works reasonably well with smaller number of exemplars.

Ablation Study.
In this experiment, we compared ADER to several simplified versions to justify our design choices. (i). ER her dinд : A vanilla exemplar replay different from ADER by using a regular L C E , rather than L K D , on exemplars. (ii). ER r andom : It differs from ER her dinд by selecting exemplars of an item at random. (iii). ER loss : It differs from ER her dinд by selecting exemplars of an item with smallest L C E . (iv). ADER equal : This version differs from ADER by selecting equal number of exemplars for each item, that is, the assumption that more frequent items should be stored more is removed. (v). ADER f ix : This version differs from ADER by not using the adaptive λ t in Eq. (4), but a fixed λ.
Comparison results are presented in Table 4, and several observations can be noted: (1). Herding is effective to selected exemplars, indicated by the better performance of ER her dinд over ER r andom and ER loss . (2). The distillation loss in Eq. (2) is helpful, indicated by the better performance of three versions of ADER over three vanilla ER methods.
(3). Selecting exemplars proportional to item frequency is helpful, indicated by the better performance of ADER over ADER equal . (4). The adaptive λ t in Eq. (2) is helpful, , indicated by the better performance of ADER over ADER f ix .

CONCLUSION
In this paper, we studied the practical and realistic continual learning setting for session-based recommendation tasks.
To prevent the recommender forgetting user preferences learned before, we propose ADER by replaying carefully chosen exemplars from previous cycles and an adaptive distillation loss. Experiment results on two widely used datasets empirically demonstrate the promising performance of ADER. Our work may inspire researchers working from similar continual learning perspective for building more robust and scalable recommenders.