Patent retrieval: a literature review <s> NTCIR collections <s> This paper describes the Patent Retrieval Task in the Fourth NTCIR Workshop, and the test collections produced in this task. We perform the invalidity search task, in which each participant group searches a patent collection for the patents that can invalidate the demand in an existing claim. We also perform the automatic patent map generation task, in which the patents associated with a specific topic are organized in a multi-dimensional matrix. <s> BIB001 </s> Patent retrieval: a literature review <s> NTCIR collections <s> In the Sixth NTCIR Workshop, we organized the Patent Retrieval Task and performed three subtasks; Japanese Retrieval, English Retrieval, and Classification. This paper describes the Japanese Retrieval Subtask and English Retrieval Subtask, both of which were intended for patent-to-patent invalidity search task. We report the evaluation results of the groups participating in those subtasks. <s> BIB002 </s> Patent retrieval: a literature review <s> NTCIR collections <s> Sixteen years ago, two of us published the first detailed empirical look at patent litigation. In this Article, we update and expand the earlier study with a new hand-coded data set. We evaluate all substantive decisions rendered by any court in every patent case filed in 2008 and 2009 — decisions made between 2009 and 2013. We consider not just patent validity but also infringement and unenforceability. Moreover, we relate the outcomes of those cases to a host of variables, including variables related to the parties, the patents, and the courts in which those cases were litigated. The result is a comprehensive picture of the outcomes of modern patent litigation, one that confirms conventional wisdom in some respects but upends it in others. In particular, we find a surprising amount of continuity in the basic outcomes of patent lawsuits over the past twenty years, despite rather dramatic changes in who brought patent suits during that time. <s> BIB003
The Japanese National Institute of Informatics Testbeds and Community for Information access Research project  (NTCIR) started in 1997 to support research in IR and other areas, focusing on CLIR. NTCIR has been organizing a series of workshops providing test collections to researchers for evaluating their methodologies on multiple CLIR tasks NTCIR  . Between NTCIR-3 and NTCIR -11 (2002-2013) , there has been dedicated tasks for patent data analysis including patent retrieval BIB001 , classification, mining, and translation. NTCIR-3: the PR task in NTCIR-3 targeted the "technology survey" problem. The dataset for this task includes: (1) full text of Japanese patent applications between (1998 and 1999), BIB003 abstract of Japanese patent applications between (1995 and 1999) along with their respective English translations, and (3) 30 search topics where each topic includes a related newspaper article. The task is to retrieve patents relevant to news articles. Both cross-genre experiments in which patents were retrieved by a newspaper clip as well as ordinary ad hoc retrieval of patents by topics were conducted BIB001 . NTCIR-4: two PR tasks were organized in NTCIR-4 BIB001 : (1) patent map generation and (2) invalidity search. The dataset for the PR tasks includes: (1) unexamined Japanese patent applications published between (1993 and 1997) along with English translations of the abstract, and (2) 34 search topics where each topic is a claim of a rejected patent application which was invalidated because of existing prior art. Relevance judgments were individual patents that can invalidate a topic claim by its own or in conjunction with other patents. Relevant passages to the invalidated claim were also annotated and added to the relevance judgments. NTCIR-5: two PR tasks were organized in NTCIR-5  : (1) document retrieval (invalidity search), and (2) patent passage retrieval. The dataset for the invalidity search task includes: (1) unexamined Japanese patent applications published between (1993 and 2002) along with English translations of the abstract, and (2) 1200 search topics where each topic is a claim of an invalidated patent application. Relevance judgments were generated in a manner similar to the one used in NTCIR-4 invalidity search task. NTCIR-6: two PR tasks were organized in NTCIR-6 BIB002 : (1) Japanese retrieval (invalidity search), and (2) English retrieval. The dataset for the Japanese retrieval task is the same one used in NTCIR-5, but more topics were used (1685 topics). The English retrieval task was focusing on finding all the citations cited by the applicant and the examiner. The dataset for these tasks includes: (1) granted patents from the USPTO between (1993 and 2000) and BIB003 3221 search topics where each topic is a granted patent published between (2000 and 2001).