Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns. <s> BIB001 </s> Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> User experience in modern content discovery applications critically depends on high-quality personalized recommendations. However, building systems that provide such recommendations presents a major challenge due to a massive pool of items, a large number of users, and requirements for recommendations to be responsive to user actions and generated on demand in real-time. Here we present Pixie, a scalable graph-based real-time recommender system that we developed and deployed at Pinterest. Given a set of user-specific pins as a query, Pixie selects in real-time from billions of possible pins those that are most related to the query. To generate recommendations, we develop Pixie Random Walk algorithm that utilizes the Pinterest object graph of 3 billion nodes and 17 billion edges. Experiments show that recommendations provided by Pixie lead up to 50% higher user engagement when compared to the previous Hadoop-based production system. Furthermore, we develop a graph pruning strategy at that leads to an additional 58% improvement in recommendations. Last, we discuss system aspects of Pixie, where a single server executes 1,200 recommendation requests per second with 60 millisecond latency. Today, systems backed by Pixie contribute to more than 80% of all user engagement on Pinterest. <s> BIB002 </s> Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> In most sponsored search platforms, advertisers bid on some keywords for their advertisements (ads). Given a search request, ad retrieval module rewrites the query into bidding keywords, and uses these keywords as keys to select Top N ads through inverted indexes. In this way, an ad will not be retrieved even if queries are related when the advertiser does not bid on corresponding keywords. Moreover, most ad retrieval approaches regard rewriting and ad-selecting as two separated tasks, and focus on boosting relevance between search queries and ads. Recently, in e-commerce sponsored search more and more personalized information has been introduced, such as user profiles, long-time and real-time clicks. Personalized information makes ad retrieval able to employ more elements (e.g. real-time clicks) as search signals and retrieval keys, however it makes ad retrieval more difficult to measure ads retrieved through different signals. ::: ::: To address these problems, we propose a novel ad retrieval framework beyond keywords and relevance in e-commerce sponsored search. Firstly, we employ historical ad click data to initialize a hierarchical network representing signals, keys and ads, in which personalized information is introduced. Then we train a model on top of the hierarchical network by learning the weights of edges. Finally we select the best edges according to the model, boosting RPM/CTR. Experimental results on our e-commerce platform demonstrate that our ad retrieval framework achieves good performance. <s> BIB003 </s> Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> Tasks such as search and recommendation have become increasingly important for E-commerce to deal with the information overload problem. To meet the diverse needs of different users, personalization plays an important role. In many large portals such as Taobao and Amazon, there are a bunch of different types of search and recommendation tasks operating simultaneously for personalization. However, most of current techniques address each task separately. This is suboptimal as no information about users shared across different tasks. In this work, we propose to learn universal user representations across multiple tasks for more effective personalization. In particular, user behavior sequences (e.g., click, bookmark or purchase of products) are modeled by LSTM and attention mechanism by integrating all the corresponding content, behavior and temporal information. User representations are shared and learned in an end-to-end setting across multiple tasks. Benefiting from better information utilization of multiple tasks, the user representations are more effective to reflect their interests and are more general to be transferred to new tasks. We refer this work as Deep User Perception Network (DUPN) and conduct an extensive set of offline and online experiments. Across all tested five different tasks, our DUPN consistently achieves better results by giving more effective user representations. Moreover, we deploy DUPN in large scale operational tasks in Taobao. Detailed implementations, e.g., incremental model updating, are also provided to address the practical issues for the real world applications. <s> BIB004 </s> Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> Recommender systems (RSs) have been the most important technology for increasing the business in Taobao, the largest online consumer-to-consumer (C2C) platform in China. There are three major challenges facing RS in Taobao: scalability, sparsity and cold start. In this paper, we present our technical solutions to address these three challenges. The methods are based on a well-known graph embedding framework. We first construct an item graph from users' behavior history, and learn the embeddings of all items in the graph. The item embeddings are employed to compute pairwise similarities between all items, which are then used in the recommendation process. To alleviate the sparsity and cold start problems, side information is incorporated into the graph embedding framework. We propose two aggregation methods to integrate the embeddings of items and the corresponding side information. Experimental results from offline experiments show that methods incorporating side information are superior to those that do not. Further, we describe the platform upon which the embedding methods are deployed and the workflow to process the billion-scale data in Taobao. Using A/B test, we show that the online Click-Through-Rates (CTRs) are improved comparing to the previous collaborative filtering based methods widely used in Taobao, further demonstrating the effectiveness and feasibility of our proposed methods in Taobao's live production environment. <s> BIB005 </s> Production Ranking Systems: A Review <s> SEPARATION OF CONCERNS <s> E-commerce sponsored search contributes an important part of revenue for the e-commerce company. In consideration of effectiveness and efficiency, a large-scale sponsored search system commonly adopts a multi-stage architecture. We name these stages as ad retrieval, ad pre-ranking and ad ranking. Ad retrieval and ad pre-ranking are collectively referred to as ad matching in this paper. We propose an end-to-end neural matching framework (EENMF) to model two tasks---vector-based ad retrieval and neural networks based ad pre-ranking. Under the deep matching framework, vector-based ad retrieval harnesses user recent behavior sequence to retrieve relevant ad candidates without the constraint of keyword bidding. Simultaneously, the deep model is employed to perform the global pre-ranking of ad candidates from multiple retrieval paths effectively and efficiently. Besides, the proposed model tries to optimize the pointwise cross-entropy loss which is consistent with the objective of predict models in the ranking stage. We conduct extensive evaluation to validate the performance of the proposed framework. In the real traffic of a large-scale e-commerce sponsored search, the proposed approach significantly outperforms the baseline. <s> BIB006
e separation of concerns across the ranking system layers has allowed organizations to create models which can approximate ideal ranking, learn query intent, perform query re-writing, and diversify retrieved results. All of these applications are addressed by training o ine models to capture relevant relationships within their representations. e necessary, o en massive computations can be performed in batch, allowing for feature spaces which encode desired relationships within the geometry of the representation space. ese relationships can then be directly leveraged by candidate selection or the online model. As the computationally heavy processes are captured in o ine representation learning and candidate selection, online models can be built as less computationally complex models, e.g. linear or shallow models, allowing for low latency response time. is separation allows for highly complex systems to be built and leveraged, but introduces failure points in the form of decreased interpretability of the ranking system behavior, di culty in tuning, and increased di culty of validation. is makes it hard to interpret experimental test results and improve upon previous iterations. Architectural decisions within the various layers of a ranking system also introduce corresponding assumptions into the overall system. O en these assumptions although necessary, can obfuscate biases and weaknesses. Ranking systems have developed two architectures which have facilitated the separation of concerns. ese are distributed technologies and one box models. As item catalogs have grown, technologies which require the housing of indices in memory are no longer feasible. As such, distributed databases which can house an index across a cluster of machines and coordinate retrieval from this index have become prevalent [16,   . ese distributed database technologies are able to house petabytes of data and run computations over their entirety. In parallel to the development of distributed computing technologies, ML models have grown in complexity, especially with the advent of deep learning. ese models require specialty hardware to support their high computational complexity, such as CUDA enabled graphics processing units (GPUs)  . O en the computations necessary for these models are infeasible to translate to distributed system frameworks, and the volume of model parameter updates required to coordinate across the clusters is prohibitive to deploying these models on distributed systems. As such, one-box architectures are a ractive. ese architectures pull processed training data from distributed data stores and train complex models in memory on a single machine BIB002  . An end to end ranking system employs both of these architectures, with distributed architectures providing a sink for raw data, embeddings, candidates and model predictions, and a source of training data and features for one-box models. is system of systems requires orchestration across frameworks, which can be handled via schedulers that are responsible for coordinating workows for training and deployment of online models  . Each set of tools employed by the layers of the ranking model must be selected with care, as increasing the number of employed tools increases the complexity of the overall ranking system. is can cause some layers to be poorly con gured, as practitioners are required to master many technologies. In some cases, separation of concerns can lead to isolation of practitioners who specialize on speci c layers within the ranking system. is can cause further poor con gurations, as practitioners may treat other layers of the system as black boxes, leading to layer speci c optimizations which may give rise to suboptimal behavior across the entire system BIB001 . We feel the separation of concerns is a useful tool, but only when employed with care. Research on productionized ranking systems tends to focus on just a single layer BIB004 BIB005 BIB006 BIB003 . Practitioners must be careful to examine the individual component layers as well as behavior across the entire system. Only the ability to interpret the system at both scales can yield an understanding of behavior and enable proper iteration to improve results. A diagram of the architectural components necessary to support a full ranking system is provided by in Figure 1 3 DATA PROCESSING