Patent retrieval: a literature review <s> Test collections and evaluation measures <s> From the Publisher: ::: This is a rigorous and complete textbook for a first course on information retrieval from the computer science (as opposed to a user-centred) perspective. The advent of the Internet and the enormous increase in volume of electronically stored information generally has led to substantial work on IR from the computer science perspective - this book provides an up-to-date student oriented treatment of the subject. <s> BIB001 </s> Patent retrieval: a literature review <s> Test collections and evaluation measures <s> Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view. <s> BIB002 </s> Patent retrieval: a literature review <s> Test collections and evaluation measures <s> Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort. <s> BIB003
As we highlighted in Sect. 3, several datasets were created to support evaluating different PR techniques. In almost all of these datasets, relevant documents to search queries were collected from the citations of topic patent documents (e.g., CLEF-IP 2009/2010/2011 collections). Because these citations represent related prior work, they are appropriate only for the related work search task. In other datasets such as CLEF-IP 2012/2013 collections, relevant documents were collected from novelty breaking citations found in examiners' search reports, and therefore, these datasets are appropriate for the patentability and invalidity search tasks although invalidity search requires the non-patent literature as well. Standard information retrieval as well as patent retrieval-specific evaluation measures are generally used to evaluate patent retrieval systems including: 1. Precision (P) and Recall (R) at top-K ranks (e.g., K = {1, 5, 10, 50, 100, 1000}). BIB001 which generally favors early retrieval of relevant documents with less focus on recall. 3. Normalized Discounted Cumulative Gain (nDCG) BIB002 which favors not only early retrieval of relevant documents but also the respective ranking quality of these documents. 4. Patent Retrieval Evaluation Score (PRES) BIB003 which was proposed specifically for recalloriented tasks such as PR. PRES focuses on the overall system recall as well as user's review effort which can be estimated from the rankings at which relevant documents are retrieved.