We implemented and evaluated a two-stage retrieval method for personalized academic search in which the initial search results are re-ranked using an author-topic profile. In academic search tasks, the user's own data can help optimizing the ranking of search results to match the searcher's specific individual needs. The author-topic profile consists of topic-specific terms, stored in a graph. We re-rank the top-1000 retrieved documents using ten features that represent the similarity between the document and the author-topic graph. We found that the re-ranking gives a small but significant improvement over the reproduced best method from the literature. Storing the profile as a graph has a number of advantages: it is flexible with respect to node and relation types; it is a visualization of knowledge that is interpretable by the user, and it offers the possibility to view relational characteristics of individual nodes.
We build on previous work with the iSearch collection. We try to reproduce the best retrieval result and use that as baseline for our experiments. In order to create an overview of previous retrieval results obtained with the collection, we first collected the set of papers that cite the iSearch test collection paper by @cite_29 (52 papers in total, according to Google Scholar, summer of 2017) and then we selected the 14 papers from that set that contain the terms isearch' and ndcg'. We excluded papers that do not report results for the iSearch collection, papers that do not report retrieval results (but results for another task, e.g. term extraction), and papers that do not report nDCG scores for the data set as a whole. After excluding those, 8 papers are left. A summary of the used methods and the highest nDCG result from each paper is listed in Table . All previous work has used Indri as index and retrieval engine.
@cite_29
1928047241
The poster discusses the characteristics needed in an information retrieval (IR) test collection to facilitate the evaluation of integrated search, i.e. search across a range of different sources but with one search box and one ranked result list, and describes and analyses a new test collection constructed for this purpose. The test collection consists of approx. 18,000 monographic records, 160,000 papers and journal articles in PDF and 275,000 abstracts with a varied set of metadata and vocabularies from the physics domain, 65 topics based on real work tasks and corresponding graded relevance assessments. The test collection may be used for systems- as well as user-oriented evaluation.