In learning to rank, one is interested in optimising the global ordering of a list of items according to their utility for users. Popular approaches learn a scoring function that scores items individually (i.e. without the context of other items in the list) by optimising a pointwise, pairwise or listwise loss. The list is then sorted in the descending order of the scores. Possible interactions between items present in the same list are taken into account in the training phase at the loss level. However, during inference, items are scored individually, and possible interactions between them are not considered. In this paper, we propose a context-aware neural network model that learns item scores by applying a self-attention mechanism. The relevance of a given item is thus determined in the context of all other items present in the list, both in training and in inference. Finally, we empirically demonstrate significant performance gains of self-attention based neural architecture over Multi-Layer Perceptron baselines. This effect is consistent across popular pointwise, pairwise and listwise losses on datasets with both implicit and explicit relevance feedback. Learning to rank (LTR) is an important area of machine learning research, lying at the core of many information retrieval (IR) systems. It arises in numerous industrial applications like search engines, recommender systems, question-answering systems, and others. A typical machine learning solution to the LTR problem involves learning a scoring function, which assigns real-valued scores to each item of a given list, based on a dataset of item features and human-curated or implicit (e.g. clickthrough logs) relevance labels. Items are then sorted in the descending order of scores [19] . Performance of the trained scoring function is usually evaluated using an Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). The Web Conference, April, 2020, Taipei, Taiwan © 2019 Copyright held by the owner/author(s). ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. https://doi.org/10.1145/nnnnnnn.nnnnnnn IR metric like Mean Reciprocal Rank (MRR) [29] , Normalised Discounted Cumulative Gain (NDCG) [16] or Mean Average Precision (MAP) [4] . In contrast to other classic machine learning problems like classification or regression, the main goal of a ranking algorithm is to determine relative preference among a group of items. Scoring items individually is a proxy of the actual learning to rank task. Users' preference for a given item on a list depends on other items present in the same list: an otherwise preferable item might become less relevant in the presence of other, more relevant items. Common learning to rank algorithms attempt to model such inter-item dependencies at the loss level. That is, items in a list are still scored individually, but the effect of their interactions on evaluation metrics is accounted for in the loss function, which usually takes a form of a pairwise (RankNet [6] , LambdaLoss [30] ) or a listwise (ListNet [9] , ListMLE [31] ) objective. For example, in LambdaMART [8] the gradient of the pairwise loss is rescaled by the change in NDCG of the list which would occur if a pair of items was swapped. Pointwise objectives, on the other hand, do not take such dependencies into account. In this work, we propose a learnable, context-aware, self-attention [27] based scoring function, which allows for modelling of interitem dependencies not only at the loss level but also in the computation of items' scores. Self-attention is a mechanism first introduced in the context of natural language processing. Unlike RNNs [14] , it does not process the input items sequentially but allows the model to attend to different parts of the input regardless of their distance from the currently processed item. We adapt the Transformer [27] , a popular self-attention based neural machine translation architecture, to the ranking task. We demonstrate that the obtained ranking model significantly improves performance over Multi-Layer Perceptron (MLP) baselines across a range of pointwise, pairwise and listwise ranking losses. Evaluation is conducted on MSLR-WEB30K [24] , the benchmark LTR dataset with multi-level relevance judgements, as well as on clickthrough data coming from Allegro.pl, a large-scale e-commerce search engine. We provide an open-source Pytorch [22] implementation of our self-attentive context-aware ranker available at url_removed. The rest of the paper is organised as follows. In Section 2 we review related work. In Section 3 we formulate the problem solved in this work. In Section 4 we describe our self-attentive ranking model. Experimental results and their discussion are presented in Section 5. In Section 6 we conduct an ablation study of various hyperparameters of our model. Finally, a summary of our work is given in Section 7. In this work, we addressed the problem of constructing a contextaware scoring function for learning to rank. We adapted the selfattention based Transformer architecture from the neural machine translation literature to propose a new type of scoring function for LTR. We demonstrated considerable performance gains of proposed neural architecture over MLP baselines across different losses and types of data, both in ranking and re-ranking setting. These experiments provide strong evidence that the gains are due to the ability of the model to score items simultaneously. As a result of our empirical study, we observed the strong performance of models trained to optimise ordinal loss function. Such models outperformed models trained with well-studied losses like LambdaLoss or LambdaMART, which were previously shown to provide tight bounds on IR metrics like NDCG. On the other hand, we observed the surprisingly poor performance of models trained to optimise RankNet and ListMLE losses. In future work, we plan to investigate the reasons for both good and poor performance of the aforementioned losses, in particular, the relation between ordinal loss and NDCG.
Learning to rank using the Transformer architecture.
{"src": "scitldr-AIC_train", "paper_id": "BJxcAX7iYB"}