Production Ranking Systems: A Review <s> Creating A Balanced Dataset <s> Clickthrough data is a particularly inexpensive and plentiful resource to obtain implicit relevance feedback for improving and personalizing search engines. However, it is well known that the probability of a user clicking on a result is strongly biased toward documents presented higher in the result set irrespective of relevance. We introduce a simple method to modify the presentation of search results that provably gives relevance judgments that are unaffected by presentation bias under reasonable assumptions. We validate this property of the training data in interactive real world experiments. Finally, we show that using these unbiased relevance judgments learning methods can be guaranteed to converge to an ideal ranking given sufficient data. <s> BIB001 </s> Production Ranking Systems: A Review <s> Creating A Balanced Dataset <s> This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions. <s> BIB002
e Learning To Rank (LTR) framework poses ranking as a supervised learning problem BIB002 . As such, both positive and negative samples need to be inferred from user interactions. Just as with positive samples, there are numerous ways to a ribute negative samples, for example, if a user exited a video stream before nishing the entire video. ere are far more possibilities for negative samples, represented in the lack of user interactions, than positive samples. Various methods for balancing the dataset are employed. One popular method is for each positive sample de ned in the dataset only sampling one corresponding negative sample. Di erent mechanisms of negative item sampling are employed to select negative samples such that speci c relationships are re ected in balanced pairs of negative samples and positive samples. is could include choosing negative items from categories that are far away from the item, excluding highly co-viewed or co-engaged products from the negatives list  . Selecting negative interactions from only a window around a positive interaction can allow an assumption of user impression. Ranking systems have an intrinsic positional bias associated with them BIB001 . Users click on higher presented results irrespective of query relevance, leading to false positive labels. Ignoring this bias and training on naively collected data can lead to models that simply t the behavior of the existing global ranking function. e FairPairs method modi es search results in a non-invasive manner is paper was accepted to the ACM's Special Interest Group on Information Retrieval (SIGIR) workshops, 2019.