Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search

Sequential recommender systems (SRS) have become a research hotspot due to its power in modeling user dynamic interests and sequential behavioral patterns. To maximize model expressive ability, a default choice is to apply a larger and deeper network architecture, which, however, often brings high network latency when generating online recommendations. Naturally, we argue that compressing the heavy recommendation models into middle- or light- weight neural networks is of great importance for practical production systems. To realize such a goal, we propose AdaRec, a knowledge distillation (KD) framework which compresses knowledge of a teacher model into a student model adaptively according to its recommendation scene by using differentiable Neural Architecture Search (NAS). Specifically, we introduce a target-oriented distillation loss to guide the structure search process for finding the student network architecture, and a cost-sensitive loss as constraints for model size, which achieves a superior trade-off between recommendation effectiveness and efficiency. In addition, we leverage Earth Mover's Distance (EMD) to realize many-to-many layer mapping during knowledge distillation, which enables each intermediate student layer to learn from other intermediate teacher layers adaptively. Extensive experiments on real-world recommendation datasets demonstrate that our model achieves competitive or better accuracy with notable inference speedup comparing to strong counterparts, while discovering diverse neural architectures for sequential recommender models under different recommendation scenes.


INTRODUCTION
S EQUENTIAL (a.k.a. session-based) recommender systems that aim to predict new interactions based on user historical ones have attracted much attention in recent years. Particularly, with the tremendous success of deep learning, deep neural network (DNN) based sequential recommendation (SR) models have yielded substantial improvements comparing to traditional collaborative filtering (CF) [1], such as neighborhood methods [2] and shallow factorization models [3]. This is because with many hidden layers, well-designed deep models could be more powerful in capturing user dynamic interests, high-level or long-range sequential relations of user interactions. More recently, Wang et al. [4] and Chen et al. [5] revealed that deep SR models such as NextItNet [6] and SASRec [7] could be stacked in a surprised depth with over 100 layers for achieving their optimal performance. However, a real problem arises as these deep SR models go bigger and deeper; that is, the model becomes too large in parameter size, and both memory and inference costs increase sharply, making the deployment of them difficult in production systems. Thereby, we argue that compressing the heavy deep SR models into moderate-or light-weight neural networks without Lei  sacrificing their accuracy is of crucial importance for practical usage. Knowledge Distillation (KD) [8] as an effective compression technique has been recently investigated in the recommender systems domain [9], [10]. By transferring useful knowledge from a big teacher network to the student network, large deep models could be slimmed into a smaller and shallower structure without performance degradation. However, existing KD methods basically distill the teacher model into a fixed-structure student model that is manually designed in advance. This potentially limits the flexibility and scalability of the student model, especially for diverse and relatively complicated scenarios in recommender systems. For example, the optimal structure for music recommendation might be different from the optimal structure for E-commerce recommendation. Ideally, we hope to generate an adaptive student model whose optimal structure considers the specific recommendation scenarios. Inspired by the success of automated machine learning (Au-toML), we propose a novel knowledge distillation method to compress the deep sequential recommendation models, termed AdaRec. AdaRec distills the knowledge of a teacher model into a student model adaptively according to the recommendation scene based on differentiable Neural Architecture Search (NAS) [11], [12], [13]. Specifically, we devise a target-oriented knowledge distillation loss to provide search supervision for learning the architecture of student network, and a cost-sensitive loss as additional regularization to constrain the model size, which achieve a superior trade-off between recommendation effectiveness and efficiency. In addition, we leverage Earth Mover's Distance (EMD) to realize effective many-to-many layer mapping during the distillation process, enabling each intermediate layer of student to learn from any other intermediate layers of its teacher. It is worth noting arXiv:2107.07173v2 [cs.IR] 10 Apr 2022 that, our method is a generic knowledge distillation framework which can directly apply to a broad class of well-known sequential recommendation models, such as NextItNet [6], SASRec [7] and BERT4Rec [14]. Besides, with the well-designed NAS architecture, our method can distill the deep sequential recommendation models into effective smaller models with diverse neural network architectures, according to the specific recommendation scenarios.
Our contributions in this paper are fourfold: • To the best of our knowledge, we are the first to propose combining Knowledge Distillation and Neural Architecture Search in the SRS tasks so as to adaptively compress the deep sequential recommendation models according to recommendation scenes. • We devise a knowledge distillation loss based on Earth Mover's Distance (EMD) and a cost-sensitive constraint to achieve a trade-off between recommendation effectiveness and efficiency. • We verify the universality of the AdaRec framework by performing KD with three different teacher models, namely, NextItNet [6], SASRec [7] and BERT4Rec [14]. • We conduct extensive experiments on three real-world recommendation datasets with different scenarios, demonstrating that AdaRec achieves competitive or better accuracy with notable inference speedup comparing to its original teacher model. Moreover, we discover diverse neural architectures of the student model in different recommendation scenarios or tasks.

Deep Sequential Recommendation
Sequential (a.k.a. session-based) recommender systems (SRS) is an important branch in the recommendation field and has become a hotspot recently due to the wide range of application scenarios and huge commercial values. Since in this paper we focus on compressing large and deep sequential recommendation models, we only review related work regarding its advancement in deep learning (DL). Deep neural networks have achieved superior recommendation accuracy in SRS tasks. In general, these models could be classified into three categories, namely RNN-based, CNN-based and self-attention based methods. Specifically, Hidasi et al. [1] proposed GRU4Rec, which is the first RNN-based sequential recommendation model. Following this work, many extended works were proposed, which either optimized a new ranking loss [15], incorporated more context features [16], or developed more advanced data augmentation [17]. While effective, these models rely heavily on the hidden states of the entire past, which cannot take full advantage of the parallel processing resources (e.g., GPU and TPU) [6] during training. Therefore, Convolutional Neural Network (CNN) and self-attention based models are proposed to mitigate such limitations [6], [7], [14], [18]. Among them, Tang et al. [18] proposed Caser, which embeds a sequence of useritem interactions into an "image" and learn sequential patterns as local features of the image by using wide convolutional filters. Subsequently, [6] proposed NextItNet, a very deep 1D temporal CNN-based recommendation model which particularly excels at modeling long-range item sequences. In addition, self-attention based models, such as SASRec [7] and BERT4Rec [14], also showed competitive accuracy for SRS tasks as well. SASRec [7] utilized the popular self-attention mechanism to model longterm sequential semantics by encoding user's historical behaviors. Inspired by the great success of BERT [19] in NLP filed, Sun et al. [14] proposed BERT4Rec, which uses the transformer architecture and masked language model to learn bidirectional item dependencies for better sequential recommendations. In this paper, we present AdaRec by applying NextItNet, SASRec & BERT4Rec as teacher networks given their superior performance in literature. In addition, Wu et al. [20] proposed SSE-PT, a personalized transformer model which applies stochastic shared embeddings (SSE) regularization to achieve personalized user representations. With the advancement on Graph Neural Networks (GNN), GNN-based sequential models, such as SR-GNN [21], GC-SAN [22] and MA-SAN [23], have also attracted attention yielded substantial improvements in recommendation accuracy. Besides, there are some other works [24], [25], [26] that designs novel neural network modules for sequential recommendations. For instance, HGN [24], a hierarchical gating neural network, adopts a feature gating and an instance gating to determine what item features should be used for recommendation. SDM [25] integrates a multi-head self-attention module with a gated fusion module to capture both short-and long-term user preferences. HAM [26] develops hybrid associations models to further capture sequential and multi-order user-item association patterns for sequential recommendations.

Knowledge Distillation
Large and deep neural networks have achieved remarkable success in recent recommendation literature [4], [27], [28]. However, the deployment of such heavy model for real production system remains a great challenge. Knowledge Distillation (KD) [8], [29] is a representative technique for model compression and acceleration. Its basic idea is to transfer important knowledge from a big teacher network to a small student network. Specifically, Tang et al. [9] proposed the first KD technique for learning to rank problems in recommender systems. However, the work only focused on distillation on very shallow neural recommendation models while its effectiveness for deep SRS keeps largely unknown. [30] presented a general knowledge distillation framework for counterfactual recommendation with four types of distillation, namely, labelbased, feature-based, sample-based and model structure-based distillation. More recently, [31] proposed a knowledge distillation framework that forces the student network to learn from both the teacher's output and the latent knowledge stored in the teacher model.
KD-based compression have also been widely studied in other domains [28], [29], [32], [33]. Nowak et al. [34] proposed a structure compression method which involves transferring the knowledge learned by multiple layers to a single layer. Wang et al. [35] progressively performed block-wise knowledge transfer from teacher networks to student networks while preserving the receptive field. Mirzadeh et al. [36] introduced a teacher assistant to mitigate the training gap between teacher model and student model. Recently, compressing pretrained language models (e.g., BERT) with KD has attracted increasing attention, and numerous novel models are proposed to effectively distill BERT from different perspectives (e.g., embedding layer, hidden layers and prediction layer), such as PKD-BERT [32], DistilBERT [28], TinyBERT [33] and BERT-EMD [29].    The proposed AdaRec consists of two primary components: teacher model and student model. In this paper, we specify AdaRec using NextItNet [6], SASRec [7] and BERT4Rec [14] as the teacher models given their superior recommendation performance. The structures of the student models are searched based on neural architecture search techniques in a differentiable manner. Specifically, we devise a target-oriented knowledge distillation loss to provide search hints for searching the architecture of student network, and an efficiency-aware loss as search constraints for constraining the model size, which achieves a superior trade-off between effectiveness and efficiency for sequential recommendations.

Neural Architecture Search
Neural Architecture Search (NAS) that automatically discovers the network architecture, has gained increasing attention recently. Early NAS methods based on reinforcement learning [37] and evolution [38] are computationally very expensive. Recent studies significantly speed up the search and evaluation stages by architecture parameter sharing, such as ENAS [39], gradient-descent based DARTS [11], [13] and SNAS [12], and hardware-aware optimization such as AMC [40] and FBNet [41], [42]. Different from existing work, we devise a target-oriented knowledge distillation loss to provide search supervision for learning the architecture of the student network, which is a joint search of student structure and knowledge transfer under the guidance of the teacher model. To our best knowledge, we are the first to propose a combination of KD and NAS for compressing the deep sequential recommendation models.

PROBLEM DEFINITION
Given a sequence of user's historical behaviors denotes the t-th interacted item of user u, the goal of SRS is to infer the item x u t+1 that the user would like to interact with at time t + 1. Since users usually pay attention to only the first few items, the top-N items are recommended, referred to as the top-N item recommendation problem.
Deep neural networks, such as NextItNet [6], SASRec [7], BERT4Rec [14], have been proposed and deployed to many SRS applications, yielding state-of-the-art performances. However, different from many traditional collaborative filtering models [43], deep sequential recommendation models often require more hidden layers to model complex and long-term relations of user actions. Recent work in [27] showed that the state-of-the-art temporal CNN model NextItNet needs over 30 layers to reach its maximum expressive ability on some benchmark datasets. Even more, [4], [5] demonstrated by experiments that both NextItNet and SASRec [7] should be stacked with over 100 layers for achieving their best accuracy. This could lead to a large model size and high network latency in practice, bringing difficulties for the deployment of them in production systems. Therefore, in this paper, we hope to reduce the model size and accelerate the inference speed for these very deep sequential recommendation models without sacrificing their accuracy.

ADAREC
We introduce a novel scene-adaptive KD-based model compression approach with differentiable NAS, called AdaRec. Formally, suppose that a large teacher model T is trained on a target dataset D, and the architecture searching space is denoted as A. The goal of AdaRec is to automatically find a high-performing student model S from A with a smaller scale. Figure 1 illustrates the overview of the AdaRec framework. The basic idea is to transfer knowledge from a large teacher recommender model T to a small student model S adaptively subject to the specific recommendation task. In this paper, we specify AdaRec using NextItNet [6], SASRec [7] and BERT4Rec [14] learning algorithm as the teacher models given their superior recommendation performance. It is noteworthy that the "teacher" is model-agnostic and potentially applicable for any sequential recommendation model with a deep network architecture. Specifically, the network structures of the student model are automatically searched based on the NAS techniques. To this end, we devise a KD loss to provide search supervision for learning the architecture of the student network and a cost-sensitive loss as search regularization to control the model size. In this manner, our AdaRec could achieve a superior trade-off between effectiveness and efficiency for SRS tasks.
In what follows, we describe AdaRec by elaborating its teacher model, student model, the KD process and the NAS searching process.

Teacher Model
We employ the block-wise (e.g., ResNet [44]) deep networks as the teacher models given their powerful performance in literature. The general framework of the teacher model consists of the bottom embedding layer, hidden layers and the softmax layer.
In terms of the hidden layers, we use the residual blocks from NextItNet [6], SASRec [7] and BERT4Rec [14] for case study, where NextItNet is based on the dilated CNN blocks, SASRec and BERT4Rec are based on the self-attention blocks. The residual block structures are depicted in Figure 1.
4.1.0.1 NextItNet: NextItNet is composed of a stack of dilated convolutional (DC) layers, which are wrapped by a residual block structure every two layers. Specifically, each input item x u is converted into an embedding vector e u , and the user-item interaction sequence X u is thereby represented by an The embedding sequence E u is then passed into a stack of dilated convolutional layers to learn feature vector E u l which is expected to capture the longrange dependencies. Here, l represents the l-th residual block and each residual block connects two consecutive DC layers. Formally, the l-th residual block with the DC operation is formalized as: where E u l−1 and E u l are input and output of the l-th residual block considered. λ × F l (E u l−1 ) + E u l−1 is a shortcut connection by element-wise addition. Similar to [4], [5], [45], we add a learnable coefficient λ to the residual mappings F l (E u l−1 ), so that the model can stack more layers, and get better results than the standard version with λ as 1. 1 F l (E u l−1 ) represents the residual mapping, which is defined as: where ψ 1 and ψ 2 represent the casual convolution operations. LN 1 and LN 2 represent layer normalization functions. σ is the ReLU activation function. Finally, a softmax output layer is applied to predict the probability distribution for the next item x u t+1 : where W is a projection matrix, and b is a bias term. 4.1.0.2 SASRec: Similar to NextItNet, SASRec is composed of a stack of self-attention (SA) layers, which are wrapped by a residual block with a self-attention layer and a feed-forward network. Formally, the l-th residual block with the SA operation is formalized as: where E u l−1 and E u l are input and output of the l-th residual block considered. λ × H l (E u l−1 ) + E u l−1 is a shortcut connection by element-wise addition. As mentioned above, We also add a learnable coefficient λ to the residual mappings H l (E u l−1 ). H l (E u l−1 ) represents the residual mapping, which is defined as: where FFN and SA represent the feed-forward and self-attention operation, respectively. LN 1 and LN 2 represent layer normalization functions. δ is the dropout function. Finally, a softmax output layer is applied to predict the probability distribution for the next item x u t+1 . For both NextItNet+ and SASRec+, the joint probability p (X u ; Ω) of each user-item interaction sequence is computed by the product of conditional distributions over interacted items as follows: where p x u i |x u 1:i−1 ; Ω is the predicted probability for the i-th item x u i conditioned on all its previous interactions [x u 1 , . . . , x u i−1 ], and Ω is the set of parameters. 4.1.0.3 BERT4Rec: BERT [19] has shown superior performance in many NLP tasks and the recently proposed BERT4Rec [14] has successfully applied the bidirectional transformer structure to SRS tasks, which achieves state-of-the-art performance on sequential recommendations. Compared to SAS-Rec using a unidirectional (left-to-right) transformer structure to capture user's dynamic interests, BERT4Rec takes bidirectional dependencies of user's sequential behaviors into consideration, and proposes a novel Masked Language Model objective to predict the masked items in the interaction sequence. Akin to SASRec, the hidden representations of l-th layer in BERT4Rec are defined similarly as Eq. (4) and (5).
During training, BERT4Rec allows m interactions in the sequence (termed as x ∆ = [x ∆1 , . . . , x ∆m ]) to be masked (i.e., replaced with a special token "[mask]") and the original interaction sequence X u is modified toX u . The goal of the Masked Language Model objective is to predict the original ids of the masked items based solely on its left and right context, which can be formalized as: where Θ is the set of parameters.
Obviously, there is a mismatch between the training and the inference since the Masked Language Model objective aims to predict the current masked items while SRS task aims to predict the next item in the future. To address this, BERT4Rec propose to append the special token "[mask]" to the end of user's behavior interaction sequence, and then predict the next item based on the final hidden representation of this token.

Student Model
Typical model compression methods usually apply KD to transfer knowledge from the large teacher model to the manually designed student model, which rely heavily on the prior knowledge of human experts to design the structure of the student model. We perform architecture search of the student network using NAS techniques rather than assigning a fixed structure in advance. Here, we introduce a block-based micro search strategy [39] to find an optimal network architecture from the search space formed by the operation sets. The search space design is key to the final performance of the searched student model. In this study, we modularize the large search space of NAS into blocks to reduce its complexity, similar to [47]. In this way, one merely needs to search a few block structures and then repeatedly stack such blocks to form the final network architecture. This strategy avoids training each block from scratch, but forces all blocks to share structures, thereby greatly reducing the time to obtain the best performing student model from a large number of candidate networks. Specifically, the searched block denoted by α c is represented as a directed acyclic graph (DAG). Each node of the block indicates a latent state h and the edge from node i to j indicates operation o i,j that transforms h i to h j . For the k-th (k > 1) searched block, we define an input node c k−1 , and an output node c k that is obtained by attentively summarized over all intermediate nodes. Formally, let O be the set of candidate operations, and we assume there is a topological order among M intermediate nodes, i.e., o i,j ∈ O exists when i < j and j ≥ 1, the search space A is thus formalized as: 4.2.0.2 Operation Set: For all the three teacher models (NextItNet, SASRec, BERT4Rec), we adopt the same operation set to search the student network architecture. In this paper, we employ lightweight CNN-based operations as candidates given that they have shown both competitive accuracy and superior efficiency in the SRS literature, compared to RNN [1] and selfattention [7] based models. The candidate operation set O is composed of four kinds of operations: convolution, pooling, skip connection and zero operation. The convolution operations include the 1D convolution, standard convolutions (without dilation), casual dilated convolutions [48] with kernel size {3, 5}. Note that the dilated convolution is used to capture long-term dependency information. The pooling operations include max pooling and average pooling with kernel size 3. The skip operation is leveraged to construct residual connections. The zero operation helps to forget the past knowledge.

Knowledge Distillation
Specifically, we distill knowledge from the teacher model from three perspectives: the embedding layer, the hidden layers, and the prediction layer. 4.3.0.1 Embedding Layer Distillation: The prediction accuracy of the sequential recommendation model, such as Nex-tItNet, can be largely improved by increasing the embedding dimension [27]. Compressing item embedding matrices without sacrificing model performance is essential online inference speedup and parameter reduction. We define L emb as the distillation loss of the embedding layer, where it is minimized by the mean squared error (MSE) between the teacher model and the student model: where E T and E S represent the item embedding matrices of teacher and student models, respectively. W e is a learnable projection parameter. 4.3.0.2 Prediction Layers Distillation: The student model is encouraged to match the prediction ability of the teacher model by learning from the probability logits of the teacher model. We define L pred using KL divergence [49] as the distillation loss of the prediction layer: where z T and z S are probability logits after passing through the softmax layer of the teacher & student models, respectively. 4.3.0.3 Hidden Layers Distillation: Since there are different numbers of hidden layers in teacher and student, conventional one-to-one layer mapping algorithms cannot be applied. Here, we employ the Earth Mover's Distance (EMD) [50] algorithm to encourage each student hidden layer to learn from multiple teacher layers adaptively. EMD measures the distance between the teacher and student networks as the minimum cumulative cost of knowledge transfer [50].
The key insight is to view network layers as distributions, and the desired transformation should make the two distributions (i.e., teacher and student layers) close.
subject to the following constraints: After solving the above optimization problem, we obtain the optimal mapping flow F H . The Earth Mover's Distance (EMD) can be defined as the work normalized by the total flow: Finally, the hidden-layer distillation loss (termed as L hidden ) can be defined by the EMD between H T and H S : By combining the above three distillation objectives (L emb , L pred , L hidden ), we can unify the knowledge distillation loss L KD between the teacher model and the student model: model into the main objective to achieve a trade-off between recommendation effectiveness and efficiency. Specifically, we define a cost-sensitive loss by considering both the parameter size and inference time: where SIZE(·) and F LOP s(·) are the normalized parameter size and the number of floating point operations (FLOPs) for each operation. The sum of FLOPs of searched operations can be used to approximate the actual inference time of the student model.

Training Procedure
Follow the common practice, we first pre-train the teacher model and then search for the student structure under the supervision of the pre-trained teacher model. When searching the student architecture, we combine the knowledge distillation loss L KD and the cost-sensitive loss L E . Besides, we also need to incorporate the cross-entropy loss (L CE ) w.r.t ground-truth labels from the training data to assist the searching process, which is defined as: where X represents the whole user-item interaction sequences in the training data, p(x u t+1 ) is the ground truth distribution for next item prediction and p(x u t+1 ) is the prediction distribution of the searched student model.
The overall loss function is defined as follows: where γ & β are hyper-parameters that balance these loss functions. After finishing the joint search of student structure and knowledge transfer under the guidance of the pretrained teacher model, we can derive an effective, efficient and adaptive architecture as the compressed sequential model by stacking the searched block structures where g i is a random noise drawn from Gumbel(0, 1) distribution, τ is a temperature coefficient to control the discreteness of the output vectors y o . In this way, we can optimize the objectives L KD and L E directly using gradient-based optimizers by using the discrete variable argmax(y o ) in the forward pass and using the continuous vector y o in the back-propagation stage.

Experimental Datasets
We conduct extensive experiments on three real-world SRS datasets from three different domains (scenes): RetailRocket from the E-commerce domain, 30Music from the music domain [52], and MovieLens-2K from the movie domain [53]. The statistics of them are provided in Table 1.
• RetailRocket 2 contains user purchasing and clicking behaviors. We set the maximum length of each sequence t to 10 so as to investigate the recommendation performance with short-range sequences. We split the sequences longer than t into multiple sub-sequences, while the ones shorter than t are padded with zero in the beginning of each sequence, similar to [6]. • 30Music is a collection of listening and playlists data retrieved from Internet radio stations through Last.fm API 3 . We process it as a middle-range sequential dataset by extracting the latest 20 actions per user. • MovieLens-2K 4 (denoted as ML-2K) is a benchmark dataset for both standard collaborative filtering and sequential recom-

Baselines and Evaluation Metrics
To verify the effectiveness and efficiency of AdaRec, we compare it with its teacher model includingNextItNet [6], SASRec [7] and BERT4Rec [14] which have been described in Section 4.1.
In addition, we have also compare it with GRU4Rec [1] and Caser [18] for reference given that the two models are recognized as two most typical sequential recommendation baselines. Following [48], we train Caser using the data augmentation method and train GRU4Rec based on the auto-regressive method. To evaluate recommendation accuracy, we adopt three popular top-N ranking metrics, including MRR@N (Mean Reciprocal Rank), HR@N (Hit Ratio) and and NDCG@N (Normalized Discounted Cumulative Gain) [54]. Here N is set to 5 and 20 for comparison. To evaluate the computational efficiency of AdaRec, we compare its model size and inference speedup with the teacher models.

Implementation Details
We divide the user-item interaction sequence X u = [x u 1:t ] for each user u into X u train = [x u 1:t−2 ] for training, x u t−1 for validation and x u t for testing, following [7]. For the teacher model NextItNet, we set the embedding dimension d to be 256, and use dilation factors of 8 × {1, 2, 4, 8} (32 layers or 16 residual blocks). For both SASRec and BERT4Rec, we set d to be 128 given that a larger d hurts their performance because of overfitting. We use 8 self-attention blocks with four heads for SASRec and BERT4Rec according to its accuracy in the validation set. When searching the architecture for the student model, we set d to one quarter of its teacher's embedding dimension (i.e., d = 64 for NextItNet and d = 32 for SASRec and BERT4Rec), γ = 0.5, β = 8, inner nodes M = 3 and student blocks K = 4. For training AdaRec, we employ AdamW [55] to optimize the parameters (e.g., embedding  Table 2 reports the performance (i.e., MRR@N , HR@N and NDCG@N (N is set to 5 and 20), parameter size and inference speedup) of AdaRec and baseline models on the three datasets.

Overall Results
From the results, we can make the following observations. First, we observe that NextItNet, SASRec and BERT4Rec outperform GRU4Rec and Caser with substantial improvements in terms of recommendation accuracy among the three datasets, which is consistent with the previous work [6], [7], [14]. Second, AdaRec with NextItNet, SASRec and BERT4Rec as teacher models attain competitive or better recommendation accuracy than their teachers, although we do not expect AdaRec beats its teacher model in accuracy. For example, on RetailRocket and 30Music, AdaRec with NextItNet as the teacher model obtains 2.9% and 3.2% improvements over its large teacher model in terms of MRR@5. Importantly, AdaRec requires much fewer parameters and achieves notable inference speedup relative to its teachers. In addition, compared to the standard KD method [8] with equivalent model size, AdaRec with NAS techniques performs substantially better with higher inference speedup.

Cross-Scene Evaluation
In this section, we investigate the scene-adaptivity of AdaRec with different recommendation scenarios. We apply the searched student architecture from one recommendation scenario to other scenarios. For example, we denote the searched student architecture for RetailRocket (i.e., E-commerce domain) with NextItNet as the teacher model as AdaRec-RetailRocket, and apply it to 30Music (i.e., music domain) and ML-2K (i.e., movie domain). For such cross-scenario validation, we randomly initialize the weights of each searched student structure and re-train it using corresponding training data and the same teacher model to ensure a fair comparison. The results are summarized in Table 3, where we omit results using SASRec and BERT4Rec as teacher models due to similar behaviors. As clearly demonstrated along the diagonal line of Table 3, we can draw that AdaRec achieves the best performance on their original recommendation scenarios in contrast to other scenarios. This is, AdaRec is scene-adaptive since the searched student network only guarantees its optimal performance on a specific recommendation scenario.

Architecture Visualization
To better understand the basic blocks of the searched student architectures, we visualize them on the three recommendation scenarios in Figure 2. For space reason, we still only show AdaRec with NextItNet as the teacher model. By comparing the searched structures for different recommendation scenarios, we can find that AdaRec for RetailRocket (from E-commerce domain) and 30Music (from music domain) are relatively lightweight, since fewer convolution operations (i.e., std cnn 3 for RetailRocket and cau cnn 3 for 30Music) are used. This is likely because the two datasets have short-range sequential dependencies. On the contrary, a more complicated student structure with diverse convolution operations(i.e., std cnn 3 and cau cnn 3 ) is learned for ML-2K so as to model the long-range dependencies. The above results well back up our claim that the proposed AdaRec is able to search adaptive student structures for different recommendation scenarios.

Ablation Studies
As described before, the loss L of AdaRec consists of three parts: the target-oriented KD loss L KD , the cost-sensitive loss L E and the standard cross-entropy loss L CE . First, we evaluate the effects of L KD and L CE by removing each of them independently, as reported in Table 4. Clearly, we find that AdaRec without each of the two losses yields sub-optimal recommendation accuracy on all three datasets. Besides, it also shows that combining distillation losses on the embedding layer L emb , prediction layer L pred and hidden layers L hidden together produces the best results. In addition, we verify the effect of the cost-sensitive loss L E by varying β, including the default case β = 8, without constraint β = 0, weak constraint β = 4 and strong constraint β = 16. The model performance and corresponding model size are illustrated in Figure 3. From the results we can see that no constraint or a small value of β lead to an increased model size; meanwhile, an aggressive β results in a smaller model size but degraded model accuracy on the other hand. An appropriate constraint (β = 8) achieves the superior trade-off between the model effectiveness and efficiency.

CONCLUSION
In this paper, we present a novel sequential recommendation knowledge distillation (KD) framework AdaRec based on the differentiable Neural Architecture Search (NAS). AdaRec compresses knowledge of large and deep sequential recommendation models into a compact student model adaptively with the recommendation scene. To the best of our knowledge, AdaRec is the first to propose combining knowledge distillation and neural architecture search in the SRS tasks so as to adaptively compress the deep sequential recommendation models according to different recommendation scenes. Besides, we devise the Earth Mover's Distance (EMD) based KD method for effective transfer of deep hidden layers and introduce a cost-sensitive constraint to achieve the trade-off between effectiveness and efficiency of SRS tasks. Extensive experiments on three real-world recommendation datasets from different scenarios demonstrate that AdaRec achieves considerably better accuracy compared to the standard KD baseline and comparable accuracy with its teacher model while accelerating inference time and reducing the computational workload largely.