Production Ranking Systems: A Review <s> Data Aggregation And Normalization <s> This paper presents an approach to automatically optimizing the retrieval quality of search engines using clickthrough data. Intuitively, a good information retrieval system should present relevant documents high in the ranking, with less relevant documents following below. While previous approaches to learning retrieval functions from examples exist, they typically require training data generated from relevance judgments by experts. This makes them difficult and expensive to apply. The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking. Such clickthrough data is available in abundance and can be recorded at very low cost. Taking a Support Vector Machine (SVM) approach, this paper presents a method for learning retrieval functions. From a theoretical perspective, this method is shown to be well-founded in a risk minimization framework. Furthermore, it is shown to be feasible even for large sets of queries and features. The theoretical results are verified in a controlled experiment. It shows that the method can effectively adapt the retrieval function of a meta-search engine to a particular group of users, outperforming Google in terms of retrieval quality after only a couple of hundred training examples. <s> BIB001 </s> Production Ranking Systems: A Review <s> Data Aggregation And Normalization <s> Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns. <s> BIB002 </s> Production Ranking Systems: A Review <s> Data Aggregation And Normalization <s> Search ranking is a fundamental problem of crucial interest to major Internet companies, including web search engines, content publishing websites and marketplaces. However, despite sharing some common characteristics a one-size-fits-all solution does not exist in this space. Given a large difference in content that needs to be ranked and the parties affected by ranking, each search ranking problem is somewhat specific. Correspondingly, search ranking at Airbnb is quite unique, being a two-sided marketplace in which one needs to optimize for host and guest preferences, in a world where a user rarely consumes the same item twice and one listing can accept only one guest for a certain set of dates. In this talk, I will discuss challenges we have encountered and Machine Learning solutions we have developed for listing ranking at Airbnb. Specifically, the listing ranking problem boils down to prioritizing listings that are appealing to the guest but at the same time demoting listings that would likely reject the guest, which is not easily solvable using basic matrix completion or a straightforward linear model. I will shed the light on how we jointly optimize the two objectives by leveraging listing quality, location relevance, reviews, host response time as well as guest and host preferences and past booking history. Finally, we will talk about our recent work on using neural network models to train listing and query embeddings for purposes of enhancing search personalization, broad search and type-ahead suggestions, which are core concepts in any modern search. <s> BIB003 </s> Production Ranking Systems: A Review <s> Data Aggregation And Normalization <s> Recommender systems (RSs) have been the most important technology for increasing the business in Taobao, the largest online consumer-to-consumer (C2C) platform in China. There are three major challenges facing RS in Taobao: scalability, sparsity and cold start. In this paper, we present our technical solutions to address these three challenges. The methods are based on a well-known graph embedding framework. We first construct an item graph from users' behavior history, and learn the embeddings of all items in the graph. The item embeddings are employed to compute pairwise similarities between all items, which are then used in the recommendation process. To alleviate the sparsity and cold start problems, side information is incorporated into the graph embedding framework. We propose two aggregation methods to integrate the embeddings of items and the corresponding side information. Experimental results from offline experiments show that methods incorporating side information are superior to those that do not. Further, we describe the platform upon which the embedding methods are deployed and the workflow to process the billion-scale data in Taobao. Using A/B test, we show that the online Click-Through-Rates (CTRs) are improved comparing to the previous collaborative filtering based methods widely used in Taobao, further demonstrating the effectiveness and feasibility of our proposed methods in Taobao's live production environment. <s> BIB004 </s> Production Ranking Systems: A Review <s> Data Aggregation And Normalization <s> E-commerce sponsored search contributes an important part of revenue for the e-commerce company. In consideration of effectiveness and efficiency, a large-scale sponsored search system commonly adopts a multi-stage architecture. We name these stages as ad retrieval, ad pre-ranking and ad ranking. Ad retrieval and ad pre-ranking are collectively referred to as ad matching in this paper. We propose an end-to-end neural matching framework (EENMF) to model two tasks---vector-based ad retrieval and neural networks based ad pre-ranking. Under the deep matching framework, vector-based ad retrieval harnesses user recent behavior sequence to retrieve relevant ad candidates without the constraint of keyword bidding. Simultaneously, the deep model is employed to perform the global pre-ranking of ad candidates from multiple retrieval paths effectively and efficiently. Besides, the proposed model tries to optimize the pointwise cross-entropy loss which is consistent with the objective of predict models in the ranking stage. We conduct extensive evaluation to validate the performance of the proposed framework. In the real traffic of a large-scale e-commerce sponsored search, the proposed approach significantly outperforms the baseline. <s> BIB005
User interaction data can be aggregated by various methods, each building its own assumptions into the ranking system. Excepting reviews and ratings, user interaction data is o en referred to as implicit feedback data. is is due to the fact that although platforms can present items to users and track speci c user interactions, the relationship between those interactions is only implied. Negative signals, indicated by lack of user interaction, can also only be implied as it is impossible to know exactly what items were viewed by the user. us, a complete set of ranking labels for all items is impossible to capture BIB001 . To reduce noise within this dataset, outliers, such as those associated with bot-like behavior or accidental clicks, are removed. is is generally done by thresholding and weighting user interactions by dwell time BIB003 BIB004 . Selection of the date range over which to populate data can also a ect models, as the volume and sparsity of data changes over the training window. Seasonal trends must also be accounted for when aggregating user interaction data. ese aggregations, although necessary, and o en speci c to dataset and application, propagate assumptions through all layers of the ranking system and should be made with care BIB002 . A ributing user clicks to searches or recommendation carousels a ects both model training and evaluation   . is is not limited to sponsored advertising and a ects all ranking problems. For example, in gathering training data for search, do products associated with the search only include those clicked immediately a er the search? Or should they include items clicked with subsequent searches assuming that these searches are re nements on the original search? Should two items clicked by the same user across days be considered related or only those clicked within the same hour be considered related? Adjusting these data aggregation layers and their underlying assumptions dictates which correlations are and are not captured within the dataset. Class labels on training instances per user click based relevancy can also change depending on the assumptions made BIB005 . ese decisions can be seen as a form of data tuning which build the assumptions into the datasets used for both training, tuning and evaluating ranking systems.