Production Ranking Systems: A Review <s> Serving Deep Learning Models <s> Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 30x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestly- sized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm. <s> BIB001 </s> Production Ranking Systems: A Review <s> Serving Deep Learning Models <s> In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, optimized serving, and a model-based data understanding tool. We also showcase and describe real examples where reinforcement learning models trained with Horizon significantly outperformed and replaced supervised learning systems at Facebook. <s> BIB002 </s> Production Ranking Systems: A Review <s> Serving Deep Learning Models <s> The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers. <s> BIB003 </s> Production Ranking Systems: A Review <s> Serving Deep Learning Models <s> E-commerce sponsored search contributes an important part of revenue for the e-commerce company. In consideration of effectiveness and efficiency, a large-scale sponsored search system commonly adopts a multi-stage architecture. We name these stages as ad retrieval, ad pre-ranking and ad ranking. Ad retrieval and ad pre-ranking are collectively referred to as ad matching in this paper. We propose an end-to-end neural matching framework (EENMF) to model two tasks---vector-based ad retrieval and neural networks based ad pre-ranking. Under the deep matching framework, vector-based ad retrieval harnesses user recent behavior sequence to retrieve relevant ad candidates without the constraint of keyword bidding. Simultaneously, the deep model is employed to perform the global pre-ranking of ad candidates from multiple retrieval paths effectively and efficiently. Besides, the proposed model tries to optimize the pointwise cross-entropy loss which is consistent with the objective of predict models in the ranking stage. We conduct extensive evaluation to validate the performance of the proposed framework. In the real traffic of a large-scale e-commerce sponsored search, the proposed approach significantly outperforms the baseline. <s> BIB004 </s> Production Ranking Systems: A Review <s> Serving Deep Learning Models <s> Tasks such as search and recommendation have become increasingly important for E-commerce to deal with the information overload problem. To meet the diverse needs of different users, personalization plays an important role. In many large portals such as Taobao and Amazon, there are a bunch of different types of search and recommendation tasks operating simultaneously for personalization. However, most of current techniques address each task separately. This is suboptimal as no information about users shared across different tasks. In this work, we propose to learn universal user representations across multiple tasks for more effective personalization. In particular, user behavior sequences (e.g., click, bookmark or purchase of products) are modeled by LSTM and attention mechanism by integrating all the corresponding content, behavior and temporal information. User representations are shared and learned in an end-to-end setting across multiple tasks. Benefiting from better information utilization of multiple tasks, the user representations are more effective to reflect their interests and are more general to be transferred to new tasks. We refer this work as Deep User Perception Network (DUPN) and conduct an extensive set of offline and online experiments. Across all tested five different tasks, our DUPN consistently achieves better results by giving more effective user representations. Moreover, we deploy DUPN in large scale operational tasks in Taobao. Detailed implementations, e.g., incremental model updating, are also provided to address the practical issues for the real world applications. <s> BIB005
As research on deep learning has progressed, a push to use these complex models in real time ranking applications has been made. is is a divergence from prior architectures which employ shallow models for real time inference. Several methods have been employed to enable deep learning models to be served and respond in real time. Each has with its own assumptions and trade-o s. Deep learning training can be scaled by distributing training over multiple GPUs by leveraging recent open source tools, such as Horizon BIB002 . ese tools provide support for training on CPU, GPU and multi-GPU in one box architectures and can provide the ability to conduct training on many GPUs distributed over numerous machines. is framework requires GPU servers, which can be cost prohibitive to purchase and maintain. Other platforms allow for distributed embeddings by adopting a parameter server (PS) paradigm which employs a cluster of servers. Model updates are communicated across the cluster through a central parameter server. ese systems are designed with the following constraints to allow for fast training: Column-wise partitioning of feature vectors among PS shards; No transmission of word vectors across the network; PS shard-side negative sampling and computation of partial vector dot products BIB001 . Here a shard is a subset of the entire dataset which is acted upon independently by one server in the cluster. ese systems are complex to maintain and support as updates are made asynchronously to the model over each shard. Embedding layers within deep learning models are fully connected, requiring many parameters. ese computations are incredibly expensive. To allow for real time embedding of raw signals, model compression via quantization of model parameters is employed BIB003 . For example, in some layers oating point precision of weights is reduced to 8 and 16 bits. is comes with a reduction in precision, but allows for a smaller memory foot-print. In this approach representation learning must be done such that models and their hardware are co-designed. e complexity of training such a model is much higher. Di erent types of quantization are performed given the acceptable drop in precision at each layer. Each layer is individually optimized, as well as the entire graph as a whole. Reduction of model parameters can be performed by model architecture decisions as well. For example, in the case of sequential models, GRUs are chosen to learn representations instead of LSTMs as they require fewer parameters BIB004 . To allow model training to be completed in a timely manner, models with many parameters trained over large datasets can be trained incrementally, with a one-time training occurring infrequently and weight incremental updates being calculated daily BIB005 .