Personalized Context-aware Re-ranking for E-commerce Recommender Systems

Ranking is a core task in E-commerce recommender systems, which aims at providing an ordered list of items to users. Typically, a ranking function is learned from the labeled dataset to optimize the global performance, which produces a ranking score for each individual item. However, it may be sub-optimal because the scoring function applies to each item individually and does not explicitly consider the mutual influence between items, as well as the differences of users' preferences or intents. Therefore, we propose a personalized context-aware re-ranking model for E-commerce recommender systems. The proposed re-ranking model can be easily deployed as a follow-up modular after ranking by directly using the existing feature vectors of ranking. It directly optimizes the whole recommendation list by employing a transformer structure to efficiently encode the information of all items in the list. Specifically, the Transformer applies a self-attention mechanism that directly models the global relationships between any pair of items in the whole list. Besides, we introduce the personalized embedding to model the differences between feature distributions for different users. Experimental results on both offline benchmarks and real-world online E-commerce systems demonstrate the significant improvements of the proposed re-ranking model.


INTRODUCTION
Ranking is crucial in E-commerce recommender systems. e quality of the ranked list given by a ranking algorithm has a great impact on users' satisfactory as well as the revenue of E-commerce recommender systems. A large amount of ranking algorithms [5,6,8,17,22,31,36] have been proposed to optimize the ranking performance. Typically ranking in recommender system only considers the user-item pair features, without considering the local context information from other items in the list, especially by those items placed alongside [9,39]. ough pairwise and listwise learning to rank methods try to solve the problem by taking the item-pair or item-list as input, they only focus on optimizing the loss function to make be er use of the labels, e.g., click-through data. ey didn't explicitly model the mutual in uences between items in the feature space.
Some works [1,38,41] tend to model the mutual in uences between items explicitly to re ne the initial list given by the previous ranking algorithm, which is known as re-ranking. e main idea is to build the scoring function by encoding intra-item pa erns in feature space. e state-of-the-art methods for encoding the feature vectors are RNN-based, such as GlobalRerank [41] and DLCM [1]. ey feed the initial list into RNN-based structure sequentially and output the encoded vector at each time step. However, RNNbased approaches have limited ability to model the interactions between items in the list. e feature information of the previous encoded item degrades along with the encoding distance. Inspired by the Transformer architecture [23] used in machine translation, we propose to use the Transformer to model the mutual in uences between items. e Transformer structure uses self-a ention mechanism where any two items can interact with each other directly without degradation over the encoding distance. Meanwhile, the encoding procedure of Transformer is more e cient than RNN-based approach because of parallelization.
Besides the interactions between items, personalization is another important factor to be considered for re-ranking in E-commerce recommender system. Re-ranking for recommender system is userspeci c, depending on the user's preferences and intents. For a user who is sensitive to price, the price feature should be more important in the re-ranking model. Typical global ranking strategy may be not optimal as it ignores the di erences between feature distributions for each user. In addition, the interaction between items is a ect by user as well. For instance, when users are in the intention of price comparison, similar items with di erent prices tend to be more aggregated in the list. When the user has no obvious shopping intention, items in the recommendation list tend to be more diverse. erefore, we introduce a personalization module into the Transformer structure to represent user's preference and intent on items. e interaction between items in the list and user can be captured simultaneously in our personalized context-aware re-ranking model. e main contributions of this paper are as follows: • Problem. We propose a personalized re-ranking problem in recommender systems, which to the best of our knowledge, is the rst time to explicitly introduce the personalized information into re-ranking task in large-scale online system. e experimental results demonstrate the e ectiveness of introducing users' representation into list representation for re-ranking. • Model. In our personalized context-aware re-ranking approach, we employ the Transformer equipped with personalized embedding to compute representations of initial input ranking list and output the re-ranking score. e selfa ention mechanism enable us to model user-speci c mutual in uences between any two items in a more e ective and e cient way compared with RNN-based approaches. • Data. We release a large scale dataset (E-commerce Reranking dataset) used in this paper. is dataset is built from a real-world E-commerce recommender system in one of the largest online shopping mobile App. Each record in the dataset contains a recommendation list for each user with users' basic information, click-through labels and features for ranking. • Evaluation. We conducted both o ine and online experiments which show that our methods signi cantly outperform the state-of-the-art approaches. e online A/B tests show that our approach achieves higher click-through rate and more revenue for online shopping mobile App.

RELATED WORK
Our work aims to re ne the initial ranking list given by the base ranker. Among these base rankers, learning to rank is one of the widely used methods. e learning to rank methods can be classi ed into three categories according to the loss function they used: pointwise [13,25], pairwise [7,21,22], and listwise [6,8,16,22,31,36,37]. To apply the learning to rank into the recommendation system, the pointwise method uses a single item as input to predict a score for ranking. e pairwise method taking two items as a pair to classify which item should be ranked rst. e listwise method uses the whole list as input and judges the likelihood possibility of the full permutation of items. ey learn a global scoring function within which the weight of a certain feature is globally learned. However, the weights of the features should be able to aware of the interactions not only between items but also between users and items.
ey use the whole initial list as input and model the complex dependencies between items in di erent ways. [1] uses unidirectional GRU [11] to encode the information of the whole list into the representation of each item. [41] uses LSTM [20] and [4] uses pointer network [33] not only to encode the whole list information, but also to generate the ranked list using a decoder. For those methods which use either GRU or LSTM to encode the dependencies of items, the capacity of the encoder is limited. is is because the RNN network encodes the items in a sequentially way according to their positions in the initial list. us the item at the end of the initial list may have li le in uence to the item at the head of the initial list, which is not true in reality. In our paper, we use transformer-like encoder, based on self-a ention mechanism to model the interactions for any of two items in O(1) distance. Besides, for those methods which use decoder to sequentially generate the ordered list, they are not suitable for online ranking system which requires strict latency criterion. As the sequential decoder using the item selected at time t-1 as input to select the item at time t, it can not be parallelized and needs n times inferences, where n is the length of the output list. [2] propose a groupwise scoring function which can be parallelized when scoring the items, but its computation cost is high because it enumerates every possible combinations of items in the list.
Besides the above ranking and re-ranking methods, our work is inspired by the following works. [32] proposed Transformer which is widely used in machine translation as its e ectiveness in encoding high-order and long distance dependencies in sequential list compared with RNN/CNN-based approaches [35,40]. FPMC [18,28] are personalized recommendation methods which learn user embedding from user's click history. YouTubeRec [14] and Bert [15] con rm that it is feasible to use the pre-trained embeddings to enhance certain task-speci c model in practice.

FORMULATION OF RE-RANKING
In this section, we rst give some preliminary knowledge about learning to rank and re-ranking methods in E-commerce recommendation system. en we formulate the problem we aim to solve in this paper. e notation used in this paper is in Table 1.
Learning to rank (o en labelled as LTR) method is widely used for ranking in real-work system to generate an ordered list for information retrieval [21,26] and recommendation [16]. In E-commerce recommendation, the LTR method learns a global scoring function based on the feature vector of items. Having this global function, the LTR method outputs an ordered list by scoring each item in the candidate set. is global scoring function is usually formulated as the following Equation: where R is the set of all users' requests in recommendation. I r is the candidate set of items for request r ∈ R. x i represents the feature space of item i. i is the label on item i, i.e., click or not. P( i |x i ; θ ) is the predicted click probability of item i given by the model with parameters θ . is the loss computed with the label i and the corresponding estimated possibility. e goal of LTR is to nd the best parameters θ that minimize the loss function L.
However, x i is not enough to learn a good scoring function. We nd that ranking in E-commerce recommender system should consider the following information: (a) dependencies between item pairs [9,39]; (b) interactions between user and items. e dependencies between item pairs can be directly learned from the initial list S = [i 1 , i 2 , ..., i n ]. We refer this kind of information as local context information, which is labeled as [4] propose approaches to make be er use of local context information. However, few works consider the interactions between the user and items. e speci c information for user-item pairs represents the preferences of users on di erent aspects of items and can not be learned directly from initial list. In this paper, we introduce a personalized vector pv i for each item i to represent the user's preference on latent characteristics of item i. For example, some people may be sensitive to price of items while others may be sensitive to brand of items. We refer this kind of information as global context information, which is labeled as PV = [pv i 1 , pv i 2 , . . . , pv i n ] in this paper.
We propose a personalized context-aware model to re-rank the initial list by utilizing both local and global context information. Compared with the global scoring function learned by LTR, our re-ranking model seeks to learn a user-speci c scoring function which is able to weight features distinctly for di erent users. e loss function of the model can be formulated as Equation 2.
where S r is the initial list given by the ranking model.θ is the parameters of re-ranking model. Our personalized contextaware model is di erent from the typical ranking approaches in the following aspects: (a) We use a ordered list S r instead of a candidate set I r . (b) When learning the scoring function for each item i, we use the total feature matrix X of all items in the list instead of one single feature vector x i to model the mutual in uences between items. (c) We use a personalized matrix PV to learn the interactions between user and items. Note that both X and PV can be used by the existing ranking approaches to achieve be er performance.

PERSONALIZED CONTEXT-AWARE RE-RANKING MODEL
In this section, we rst give an overview of our proposed Personalized Context-aware Re-ranking Model (PCRM). en we introduce each component of our model in detail.

Model Architecture
e architecture of PCRM model is shown in Figure 1. e model consists of three parts: the input layer, the encoding layer and the output layer. It takes the initial list of items generated by LTR method as input and outputs a re-ranked list. In input layer, we utilize the interaction between user and each item i to generate a personalized vector pv i . pv i , i = i 1 , ..., i n is the personalized vector of user u on item i which represents the user's preference on latent characteristics of item i. We concat pv i with raw feature vector x i , then add it with a position embedding pe i to get the input vector for encoding layer. e encoding layer is a Transformer-like encoder made up of multi-blocks, each of which contains a multihead a ention module and a feed forward network (abbreviated as FFN), the detailed structure is shown in Figure 1 (a). A er encoding layer, the model generates a score for each item using a so max function. e detailed structure will be introduced separately in the following sections.

Input Layer
e function of this layer is to make a comprehensive representation of all input features and feed it to the next layer. First we have a xed length of initial sequential list S = [i 1 , i 2 , ..., i n ] given by the learning to rank method. Same as the learning to rank method, we have a feature matrix X ∈ R n×d feature . Each row in X represents the raw feature vector x i for each item i ∈ S.
Personalized Vector. As shown in Figure 1 (b), we concat the raw feature matrix X ∈ R n×d feature with a personalized matrix PV ∈ R n×d pv to get the intermediate embedding matrix E ∈ R n×(d feature +d p ) , which is shown in Equation 3. PV is produced by a pre-trained model which will be introduced in the following section.
Position Embedding. In order to utilize the sequential information in the initial list, we inject a position embedding PE ∈ R n×(d feature +d p ) into the input embedding. en the embedding matrix for encoding layer can be calculated using Equation 4. In this paper, a learnable PE is used which we found that it led to slightly outperform the xed position embedding used in [32].
At last we use one simple feed-forward network to convert the . . .    .

Encoding Layer
e function of encoding layer in Figure 1(a) is to learn a highlevel representation for item list S by combining both local and global context information. Transformer [32] has been proved to be e ective in many NLP tasks, specially machine translation for its powerful encoding and decoding ability compared to RNN-based approaches [11,12,20]. Self-a ention mechanism in Transformer is particularly suitable in our re-ranking task as it directly models the mutual in uence for any two items regardless the distance between them. Without the decay with distance, Transformer can capture more interactions between items which are far away from each other in the initial list. As shown in Figure 1(b), our encoding module consists of N x blocks. Each block (in Figure 1(a)) contains a multi-head a ention layer and Point-wise Feed-Forward Network (FFN) layer.
Attention Module. e a ention function we used in this paper is de ned as Equation 6: where matrices Q, K, V represent queries, keys and values respectively. d K is the dimensionality of matrix K to avoid large value of the inner product. so max is used to convert the value of innerproduct into the adding weight of the value vector V . In our paper, we use self-a ention where Q, K and V are projected from the same matrices. Each row in E is corresponding to each item in the ranking list. E contains not only the features of items but also the interaction between user and items.
In our model, any two items can a ention with each other. e self-a ention result of each item is the weight sum of the all the items in initial list. Compared with Language Model [27] and SASREC [23], which use the masked self-a ention by preventing the a erward item i n+1 from being encoded into the representation of the item i n .
is is because that [23] focuses on next bucket recommendation: given a sequence of clicked items by the user from 0 th time slot to t-1 th time slot, the algorithm predicts which item the user will click at t th time slot. us it is not allowed to leak the information of ground truth (item at at t th time slot). However, in re-ranking se ings, the information of all the items can be used.
To model more complex mutual in uences, we use the multihead a ention as shown in Equation 7: where h is the number of headers. e in uence of di erent value of h will be studied in the ablation study in the next section. Moreover, to improve the generality of the model and make the model stable for training, we apply the dropout layer (abbreviated as dropout) and residual connections (abbreviated as RC). e dropout is proposed by [29] to address the problem of over ing. e residual connections is used to alleviate the gradients vanishing problem as it directly propagates the raw input of lower layer to higher layer and adds with the self-a ention [19] results. e RC is an essential part when we stacks multiple blocks of Transformer encoder. e ablation study in the next section will compare the performance when removing these two components separately. In total, the a ention layer can be formulated as Equation 8.
where LayerNorm is the standard normalization layer proposed by [3].
Position-wise Feed-Forward Network. e function of this position-wise Feed-Forward Network (FFN) is mainly to enhance the model with non-linearity and interactions between di erent dimensions of the input vectors. Similar to A ention module, we also use Dropout to avoid over ing and residual connection to enhance the power of the model by hierarchically learning. LayerNorm is Personalized Context-aware Re-ranking for E-commerce Recommender Systems SIGIR19, 2019, France normalization which is e ective on accelerating and stabilizing the training process. Finally, the output of the FFN layer can be calculated using Equation 9: (1) and b (2) are learnable parameters.
Stacking the Encoding Layer. Here we use a ention module followed by the position-wise FFN as a block of Transformer [32] encoder. A er encoding by one block, the output of encoding layer for item i (abbreviated as F (0) i ) has already contained the interaction between di erent items. We believe that taking F i contains more complex and high-order mutual information. In the next section, we study how many blocks is best for our task. e output of the encoding layer a er stacking N x blocks can be represented as Equation 10.
where N x is the stacking number of the encoding layer. Generally, N x is larger than 0 since it is obvious that the embedding vector would directly connect to the output layer when N x = 0. So the model reduces to a two-layer feed-forward network and the performance would greatly degrade.

Output Layer
e function of the output layer is mainly to generate a score for each item i = i 1 , . . . , i n (labeled as Score(i) in Figure 1 (b)). We use one linear layer followed by a so max layer. e output of so max layer is the probabilities of click for each item, which is labeled as P( i |X , PV ;θ ). We use P( i |X , PV ;θ ) as Score(i) to re-rank the items in O(1) time complexity. e formulation of Score(i) is shown in Equation 11.
where W F is learnable projection matrix, and b F is bias term. n is the number of items in the initial list.
In the training process, we use the click-through data as label and minimize the loss function shown in Equation 12. We update the parametersθ of the PCRM model by Adam optimizer [24].

Personalized Module
In this section, we introduce the approach to calculate the personalized matrix PV , which represents interactions between user and items. e straightforward approach is to learn PV with PCRM model in an end-to-end manner on the re-ranking loss. However, as explained in Section3, the re-ranking task is to re ne of the output of learning to rank approaches. e task-speci c representation learned on re-ranking task lacks users' global preference. erefore, we utilize a pre-trained neural network to produce user's personalized embeddings PV which are then used as features for PCRM model. e pre-trained neural network is learned from the whole click-through logs in e-commerce platform. Figure 1(c) shows the structure of pre-trained model in our paper. is sigmoid layer outputs the click probability on item i for user u (P( i |H u , u; θ )) given user's all behavior history (H u ). P( i |H u , u; θ ) is the predicted click probability. H u is the sequence of items clicked by the user u. e side information of user is also used as input, e.g., gender, age, purchasing level, etc. e loss of the model is calculated by a point-wise cross entropy function which is shown in Equation 13.
where D is the set of items displayed to user u within the whole clickthrough logs in e-commerce platform. θ is the parameter matrix of pre-trained model. i is the label (click or not) on item i. Inspired by the work [14], we employ the hidden vector before the sigmoid layer as the personalized vector pv i (in Figure 1(c)) that feeds into our PCRM model.

EXPERIMENTAL RESUTLS
In this section, we rst introduce the datasets and baselines used for evaluation. en we compare our methods with baselines on these datasets to evaluate the e ectiveness of our PCRM model. At the same time, the ablation study is conducted to help understand which part of our model contributes most to the overall performance.

Datasets
We evaluate our approach based on two datasets: Yahoo! Webscope v2.0 set 1 1 (abbreviated as Yahoo Letor dataset) and E-commerce Re-ranking dataset 2 . To the best of our knowledge, there is no publically available e-commerce re-ranking dataset with context information for recommendation. erefore, we construct E-commerce Reranking dataset from a popular e-commerce platform. e overview of two datasets are shown in Table 2.
Yahoo Letor dataset. We process the Yahoo Letor dataset to t for the ranking model of recommendation using the same method in Seq2Slate [4]. Firstly, we convert the ratings (0 to 4) to binary labels using a threshold T b . Secondly, we use a decay factor η to simulate the impression probabilities of items. All the documents in Yahoo Letor dataset are rated by the experts under the assumption that all documents for each query can be viewed by the users compeletely. However, in the real world recommendation scenario, items are viewed by the users in a top-down manner. As the screen of the mobile App can only show limit number of items, the bigger the ranked position of one item, the smaller probability of that this item can be viewed by the user. In this paper, we use 1/pos(i) η as the decay probability, where pos(i) is the ranking position of item i in the initial list and η is the hyper parameter. E-commerce Re-ranking dataset. e dataset contains a largescale records in form of click-through data from a real world recommendation system. Each record in the dataset contains a recommendation list for each user with users' basic information, click-through labels and features for ranking.

Baselines
Both learning to rank (LTR) and re-ranking methods can act as our baselines.
LTR. e LTR methods are used in two tasks. Firstly, the LTR methods can generate an initial list S r for the re-ranking model from a candidate set I r for each user request r . Secondly, the LTR methods which use pairwise or listwise loss function can act as re-ranking methods by taking the initial list S r as input and conducting the ranking algorithm for another time. e representative LTR methods used in this paper include: • SVMRank [22]: is is a representative learning to rank method which use the pairwise loss to model the scoring function. • LambdaMart [6]: is is a representative learning to rank method which use the listwise loss to model the scoring function. LambdaMart is the state-of-the-art LTR among those LTR methods equipped with the listwise loss function according to [34]'s evaluation. • DNN-based LTR: is is the learning to rank method which is deployed in our online recommender system. It use the standard Wide&Deep network structure [10] to model the scoring function via the pointwise loss function.
Re-ranking. As mentioned in the related work section, the existing re-ranking methods include DLCM [1], Seq2Slate [4] and GlobalRerank [41]. DLCM [1] and GlobalRerank [41] focus on reranking in information retrieval. Seq2Slate [4] focuses on re-ranking in both recommendation and information retrieval. In this paper, we only choose DLCM as baseline method. Seq2Slate and GlobalRerank are not chosen as baselines because they all use the decoder structure to generate the re-ranked list. Seq2Slate uses pointer network to generate re-ranked list sequentially. GlobalRerank uses RNN equipped with a ention mechanism as the decoder. e decoder structure outputs the item one by one. Whether an item is selected depends on the items which are chosen before it. As a consequence, both Seq2Slate and GlobalRerank can not be parallelized in online inference. e time complexity for Seq2Slate and GlobalRerank at interference phase is O(n) × RT , where n is the length of the initial list and RT is the time for a single ranking or re-ranking request. e latency for re-ranking by Seq2Slate and GlobalRerank is unacceptable because of the strict latency criterion for online recommender service.
• DLCM [1]: It is a re-ranking model used in information retrieval based on the initial list generated by LTR methods. e GRU is used to encode the local context information into a global vector. Combing the global vector and each feature vector, it learns a more powerful scoring function than the global ranking function of LTR.

Evaluation Metrics
For o ine evaluation, we use Precision and MAP to compare di erent methods. More speci cally, we use Precision@5, Precision@10 for precision and MAP@5, MAP@10 and MAP@30 for MAP. As the maximum length of initial list in our experiments is 30, MAP@30 represents total MAP and is denoted by MAP in this paper. e de nitions of the metrics are as follows.
Precision@k is de ned as the the fraction of clicked items in the top-k recommended items for all test samples, as shown in Equation 14.
where R is the set of all user requests in the test dataset. S r is the ordered list of items given by the re-ranking model for each request r ∈ R and S r (i) is the i-th item. 1 is the indicator function whether item i is clicked or not.
MAP@k is short for the mean average precision of all ranked lists cut o by k in the test dataset. It is de ned as follows.
For online A/B tests, we use PV, IPV, CTR and GMV as metrics. PV and IPV are de ned as the total number of items viewed and clicked by the users. CTR is the clickthrough rate and can be calculated by IPV/PV. GMV is the total amount of money (revenue) user spent on the recommended items.

Experimental Settings
Training Dataset. For both Yahoo Letor dataset and E-commerce Re-ranking dataset, the length of the initial list given by LTR is 30. When preprocessing the Yahoo Letor dataset for recommendation, the decay parameter η = 0.2 and threshold T b used to generate binary labels is 1.5. Same number of features are used by all the algorithms on each dataset to ensure the fairness of comparison.
Model. For both baselines and our PCRM model, we use the same value for those critical hyper parameters. e hidden dimensionality d model is set to 1024 for Yahoo Letor dataset and 64 for E-commerce Re-ranking dataset. e learning rate of Adam optimizer in our PCRM model is the same with [32]. Negative log likelihood loss function is used as shown in Equation 12. p dropout is set to 0.1. e batch size is set to 256 for Yahoo Letor dataset and 512 for E-commerce Re-ranking dataset. We use 4 blocks of transformer with 3 multi-heads. We also tried di erent number of blocks and multi-heads to see the performance variances by ablation study. e rest se ings belonging to the customized parts of our model will be listed at the corresponding parts in the evaluation section.

O line Experiments
In this section, we rst conduct o ine evaluations on Yahoo Letor dataset and E-commerce Re-ranking dataset. en we show the results of online A/B tests. We also conduct the ablation study to help nding which part of our PCRM model contributes most to the performance.

O line Evaluation on Yahoo Letor dataset.
In this section, we conduct evaluation on Yahoo Letor dataset to discuss the following questions: • Q0: Does our PCRM model outperform the state-of-the-art methods and why? • Q1: Does the performance vary according to initial lists generated by di erent LTR approaches?
e evaluation results are shown in Table 3. We compare the baselines and our PCRM-BASE model based on two di erent initial lists which are generated by LambdaMART and SVMRank respectively. PCRM-BASE is the variant of our PCRM model without the personalized module. Note that Yahoo Letor dataset does not contain user-related information, thus we only conduct PCRM-BASE for comparison. SVMRank and LambdaMart are also used for reranking. For SVMRank, we use the implementation in [22]. For LambdaMart, we use the implementation from RankLib 3 . Table 3 shows that our PCRM-BASE achieves stable and signi cant performance improvements comparing with all baselines. When based on the initial list generated by SVMRank, PCRM-BASE outperforms DLCM by 1.7% at MAP and 1.6% at Precision@5. e gap gets larger when comparing with SVMRank which has 5.6% increase at MAP and 5.7% increase at Precision@5. When based n the initial list generated by LambdaMART, PCRM-BASE outperforms DLCM by 1.1% at MAP and 0.2% at Precision@5. PCRM-BASE also achieves 3.1% improvements on MAP and 1.3% improvements on Precision@5 comparing with LambdaMART .
PCRM-BASE uses the same training data as DLCM and does not contain the personalized module. e performance gain over DLCM mainly comes from the powerful encoding ability of Transformer. Multi-head self-a ention mechanism is more good at modeling mutual in uence between two vectors, especially when the length of encoding list gets longer [23]. In our model, the a ention mechanism can model the interactions of any item-pairs in O(1) encoding distance.
As PCRM-BASE uses the Transformer-like structure, there are many sub-modules which may contribute to the performance gain. We conduct the ablation study to help us understand which subdesign helps the most to beat the baselines. e ablation study is conducted on the initial list generated by SVMRank. Similar results were found when using the initial list generated by LambdaMART and we omits the results in this paper as space is limited. Table 4 show the results of ablation in three parts: e rst part ( rst row) shows the performance of the baseline DLCM. e second part (second row) "Default" is the best performance of our PCRM-BASE model. e third part (the rest rows) shows di erent ablation variants of our PCRM model which include: remove position embedding (PE), remove residual connection (RC), remove dropout layer, use di erent number of blocks and use di erent number of 3 h ps://sourceforge.net/p/lemur/wiki/RankLib/ heads in multi-head a ention. Note that we set b=4 and h=3 in our "Default" PCRM model.
As shown in Table 4, the performance of our model degrades greatly a er removing position embedding. is con rms the importance of sequential information given by the initial list. By removing the position embedding, our model learns the scoring function from the candidate set instead of an ordered list. Note that even without position embedding, our PCRM-BASE still achieve comparable performance with DLCM, which further con rms that our PCRM-BASE model can encode the initial list more e ectively than DLCM.
the MAP of our model slightly decreases by 0.1% and 0.7% respectively when removing residual connections and dropout layer, which indicates that our model is less severe to the problems such as gradients vanishing and over ing.
e performance of our model rst increases with the number of blocks (1-¿2-¿4) and decreases a erwards (4-¿6-¿8), as over ing happens when we stack 8 encoding blocks together.
We also tried di erent se ings(h = 1, 2, 3, 4) in the multi-head a ention layer. No signi cant improvements are observed in Table 4, which is di erent from the conclusions derived from NLP tasks [30]. e experiments in NLP show that when using more heads in multi-head a ention mechanism, it is usually helpful since more information can be captured for the following reasons. (1) From Equation 7 we nd that the function of each head is playing a role of mapping the original feature vector into a di erent subspace.
us using more heads, we can model more interactions of items in di erent sub-spaces. (2) [30] indicates that using more heads is helpful in encoding the information of long sequence. is is resonable because the output vector for a certain item is the weighted sum of all item vectors in the list. When the sequence becomes longer, each item in the list contributes less to the output vector. However, in our re-ranking se ings, all the items in the initial list are highly homogenous and the average length of the initial list is 18. ere are minor improvements when mapping the original feature vector into di erent subspaces. Besides, our sequence is not long enough to use multiple heads. As a consequence, we suggest to use only one head to save computation costs because the performance improvements are not obvious.

O line
Evaluation on E-commerce Re-ranking dataset. We conduct the o ine evaluation on E-commerce Re-ranking dataset to answer the following question.
• Q2: What is performance of our PCRM model equipped with personalized module?
e evaluation results are shown in Table 5. We compare different variant PCRM model with DLCM baseline. For our PCRM models, we not only Waeluatethe performance of PCRM-BASE, but also evaluate the performance of the variant of model equipped with the pre-trained personalized vector PV , which is labelled as PCRM-Personalized-Pretrain. do not compare wevith SVMRank and LambdaMART because the public released implementations can not handle our large dataset and out-of-memory error happens. As our previous evaluation on Yahoo Letor dataset already con rms that DLCM achieve be er performance in all metrics and [1] has consistent results, we omit the comparison with SVMRank and LambdaMART on our E-commerce Re-ranking dataset. e initial  list is generated by a DNN-based LTR method which is deployed in our real world recommender system. All the re-ranking methods listed in Table 5 use the same training dataset, which is also the same with DNN-based LTR method. e rst row in Table 5 shows the performance of the DNN-based LTR without any re-ranking methods. e numbers in the brackets are the related improvement compared with DNN-based LTR without re-ranking. Table 5 shows consistent results with Table 3 when comparing with PCRM-BASE and DLCM. Our PCRM-BASE outperforms DLCM by 2.34% at MAP and 4.1% at Precision@5. Recall that on Yahoo Letor dataset, PCRM-BASE achieves 1.7% improvements on MAP and 1.6% improvements on Precision@5. e performance gain on our E-commerce Re-ranking dataset is much larger than on Yahoo Letor dataset. is is highly related with the properties of Yahoo Letor dataset. As mentioned in the previous section, in order to simulate the training data in recommendation, we convert the ratings ({0,1,2,3,4}) into binary label (relevant and irrelevant) to simulate click-through data. Our statistics of the converted Yahoo Letor dataset show that the average click through rate is 30%, which mean that for each query with 30 recommended documents, about 9 documents are clicked by the users. However, the average clickthrough rate in our real world E-commerce Re-ranking dataset is no more than 5%. It means that ranking on Yahoo Letor dataset is much easier than on E-commerce Re-ranking dataset. is is also con rmed by the value of MAP for the same ranking methods on two datasets: DLCM can achieve 0.64 MAP on Yahoo Letor dataset but can only achieve 0.28 MAP on E-commerce Re-ranking dataset. Combining Table 5 and Table 3, we nd that the harder the ranking task, the larger improvements of our model. Table 5 shows that our PCRM-Personalized-Pretrain achieves signi cant performance improvements comparing with PCRM-BASE. PCRM-Personalized-Pretrain outperforms PCRM-BASE by 4.5% at MAP and 6.8% at Precision@5. is is mainly imported by the global context information PV , which is learned by a pre-trained model whose architecture is illustrated in Figure 1 (c). PCRM-Personalized-Pretrain has three strengths: (1) e pre-trained model can use longer period of logs which is helpful in learning long-term user interest. (2) e pre-trained model can use logs from the whole e-commerce platform, which is not limited in the recommendation logs. (3) e personalized embeddings learned by pre-trained is more generic to represent user's preference on items.

Online Experiments
We also conduct online A/B tests at a real world e-commerce recommender system on online metrics which includes PV, IPV, CTR and GMV. e meaning of these metrics is explained in the previous "Evaluation Metrics" section. ese metrics evaluate how much willingness for users to view (PV), click (IPV, CTR) and purchase (GMV) in a recommender system. e online A/B tests last for one day. For each algorithm, there are hundred thousand users and millions of requests for online test. Table 6 shows the relative improvements of three methods to an online base ranker. Firstly, the online A/B test shows that using re-ranking helps increasing the online metrics no ma er what kinds of re-ranking methods are. Again, we can conclude that re-ranking helps improving the performance by considering the local context information. It is note that 0.7% increase on PV is signi cant in our large-scale online system because it means that about billions of extra items is viewed by the users a er using the re-ranking method. Secondly, we can conclude that our PCRM-BASE model brings an extra 0.5% increases on viewed items and extra 2.44% increases on clicked items and extra 0.23% increases on the total GMV. Lastly, by using the personalized module, our PCRM-Personalized-Pretrain can further improve the GMV by 6.31%. Recall that in o ine experiments on E-commerce Re-ranking dataset, PCRM-Personalized-Pretrain has 4.5% extra increases compared with PCRM-BASE. e result shows that personalization with pre-trained user's representation can a ect the interaction of item-item, and bring signi cant gain for re-ranking.

Visualizing Attention Weights
We visualize the a ention weights learned by our model to answer the following question.
• Q3: Can self-a ention learn meaningful information in respect to di erent aspects, for example, positions and characteristics of items?
Attention on Characteristics. We rst visualize the average a ention weights between items on two characteristics: category and price. e results caculated on the test dataset are shown in Figure 2. Each block in the heatmap represents the average a ention weights between items belonging to seven main cateogries. e darker the block, the larger the weight. From Figure 2(a) we can conclude that the a ention mechanism can successfully capture mutual-in uence in di erent categories. e items with similar categories tend to have larger a ention weights, indicating larger mutual in uences. For example, "men's shoes" has more in uences on "women's shoes" than on "computer". It is also easy to understand that "computer", "mobile phone" and "home appliance" have large a ention weights with each other because they are all electronics. Similar cases can be observed in Figure 2(b). In Figure 2(b), we classify the items into 7 levels according to their prices. e closer price between items, the larger the mutual in uences.
Attention on Positions. e visualization of average a ention weights on positions in the initial list is shown in Figure 3. Firstly, Figure 3(a) showed the self-a ention mechanism in our model can capture the mutual in uences regardless of the encoding distances as well as the position bias in recommendation list. Items ranked ahead of the list usually is more likely to be clicked and thus have more in uences on those items at the tail of the list. For example, we observe that items at the rst position have larger impacts on items at 30th position than those items at 26th position even though      the la er is more closer to it. e e ect of position embeddings is also obvious compared with the Figure 3(b), whose a ention weights between each position are more uniformally distributed.

CONCLUSION AND FUTURE WORK
In this paper, we proposed a personalized context-aware re-ranking model (PCRM) to re ne the initial list given by state-of-the-art learning to rank methods. e introduction of personalized information into the re-ranking model can bring signi cant performance improvements. In the re-ranking model, we used Transformer network to encode both the dependencies among items and the interactions between the user and items. Both the online and o ine experiments demonstrated that our PCRM model can greatly improve the ranking performance on either public benchmark dataset or our released dataset. In addition, the new released dataset can help researchers to optimize the ranking algorithms for e-commerce recommendation.
Our work explicitly models the complex item-item relationships in feature space. We believe that optimization in label space can also helps. e pairwise or listwise loss function aims to dig more information in label space. However, limited improvements are achieved in our experiments for re-ranking. e labels of clickthrough data in the paper is click or not. It is interesting to construct more partial ordering relations (e.g. order by stay time) in clickthrough data. Another future direction is learning to diversify by re-ranking. Even thougth our model does not hurt the ranking diversity in practice. It is worth to try to introduce the goal of diversi cation into our re-ranking model. We will further explore this direction in the future work.