Production Ranking Systems: A Review <s> Datasets <s> This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions. <s> BIB001 </s> Production Ranking Systems: A Review <s> Datasets <s> YouTube represents one of the largest scale and most sophisticated industrial recommendation systems in existence. In this paper, we describe the system at a high level and focus on the dramatic performance improvements brought by deep learning. The paper is split according to the classic two-stage information retrieval dichotomy: first, we detail a deep candidate generation model and then describe a separate deep ranking model. We also provide practical lessons and insights derived from designing, iterating and maintaining a massive recommendation system with enormous user-facing impact. <s> BIB002 </s> Production Ranking Systems: A Review <s> Datasets <s> The explosive popularity of e-commerce sites has reshaped users» shopping habits and an increasing number of users prefer to spend more time shopping online. This evolution allows e-commerce sites to observe rich data about users. The majority of traditional recommender systems have focused on the macro interactions between users and items, i.e., the purchase history of a customer. However, within each macro interaction between a user and an item, the user actually performs a sequence of micro behaviors, which indicate how the user locates the item, what activities the user conducts on the item (e.g., reading the comments, carting, and ordering) and how long the user stays with the item. Such micro behaviors offer fine-grained and deep understandings about users and provide tremendous opportunities to advance recommender systems in e-commerce. However, exploiting micro behaviors for recommendations is rather limited, which motivates us to investigate e-commerce recommendations from a micro-behavior perspective in this paper. Particularly, we uncover the effects of micro behaviors on recommendations and propose an interpretable Recommendation framework RIB, which models inherently the sequence of mIcro Behaviors and their effects. Experimental results on datasets from a real e-commence site demonstrate the effectiveness of the proposed framework and the importance of micro behaviors for recommendations. <s> BIB003 </s> Production Ranking Systems: A Review <s> Datasets <s> E-commerce sponsored search contributes an important part of revenue for the e-commerce company. In consideration of effectiveness and efficiency, a large-scale sponsored search system commonly adopts a multi-stage architecture. We name these stages as ad retrieval, ad pre-ranking and ad ranking. Ad retrieval and ad pre-ranking are collectively referred to as ad matching in this paper. We propose an end-to-end neural matching framework (EENMF) to model two tasks---vector-based ad retrieval and neural networks based ad pre-ranking. Under the deep matching framework, vector-based ad retrieval harnesses user recent behavior sequence to retrieve relevant ad candidates without the constraint of keyword bidding. Simultaneously, the deep model is employed to perform the global pre-ranking of ad candidates from multiple retrieval paths effectively and efficiently. Besides, the proposed model tries to optimize the pointwise cross-entropy loss which is consistent with the objective of predict models in the ranking stage. We conduct extensive evaluation to validate the performance of the proposed framework. In the real traffic of a large-scale e-commerce sponsored search, the proposed approach significantly outperforms the baseline. <s> BIB004
Increasing user interactions is the primary goal of ranking systems BIB002 BIB001 . As such, user interaction logs captured by the platform o en serves to be the richest source of training data for the system. Some user interactions are ubiquitous, such as user clicks or user item ratings. Others are platform speci c, such as purchases on e-commerce platforms, duration of viewing time on streaming content platforms, or likes on social media platforms. Data about the items themselves, such as title, category, associated text and cost, is referred to as side information. Side information is gathered by platforms either by user feedback, such as tagging on social media platforms, or by content providers, such as a ribute labels provided by vendors on e-commerce platforms, or by the platform itself such as item categories. Most data used by ranking systems are sparse high dimensionality vocabularies. is is especially true for user interaction data, where each item in a catalog can be seen as a word in the vocabulary with few user interactions BIB003 . Side information contains a mix of sparse and dense data, such as sparse multi-hot encodings of text, or dense data of item price and size or item images. Prior to representation learning and model training, raw data must be normalized, featurized and forma ed. To reduce noise and reduce computation time, large cardinality spaces can be trimmed via thresholding to drop highly sparse dimensions from the vocabulary. Out of vocabulary components can either be dropped or mapped to a default null e query can be items, user provided query strings or users in generalized ranking approach, and thus query processor can either act as a query re-writer, user personalization, or an item diversi cation layer. representation BIB004 . Dense features which contain high variability or follow an exponential distribution can be normalized and/or smoothed prior to ingestion by models.