Let @math @math be a given set of @math string documents of total length @math , our task is to index @math , such that the @math most relevant documents for an online query pattern @math of length @math can be retrieved efficiently. We propose an index of size @math bits and @math query time for the basic relevance metric , where @math is the size (in bits) of a compressed full text index of @math , with @math time for searching a pattern of length @math . We further reduce the space to @math bits, however the query time will be @math , where @math is the alphabet size and @math is any constant.
Table gives a summary of the major results in the top- @math frequent document retrieval problem. The time complexities are simplified by assuming that we are using the full text index proposed by Belazzougui and Navarro, of size @math bits and @math , where @math is the @math th order empirical entropy of @math @cite_2 . We also assume @math for some @math and @math is any constant.
@cite_2
2951300344 2737269238 1917123765 1975998118
Let @math @math be a given set of @math string documents of total length @math , our task is to index @math , such that the @math most relevant documents for an online query pattern @math of length @math can be retrieved efficiently. We propose an index of size @math bits and @math query time for the basic relevance metric , where @math is the size (in bits) of a compressed full text index of @math , with @math time for searching a pattern of length @math . We further reduce the space to @math bits, however the query time will be @math , where @math is the alphabet size and @math is any constant. We consider document listing on string collections, that is, finding in which strings a given pattern appears. In particular, we focus on repetitive collections: a collection of size @math over alphabet @math is composed of @math copies of a string of size @math , and @math edits are applied on ranges of copies. We introduce the first document listing index with size @math , precisely @math bits, and with useful worst-case time guarantees: Given a pattern of length @math , the index reports the @math strings where it appears in time @math , for any constant @math (and tells in time @math if @math ). Our technique is to augment a range data structure that is commonly used on grammar-based indexes, so that instead of retrieving all the pattern occurrences, it computes useful summaries on them. We show that the idea has independent interest: we introduce the first grammar-based index that, on a text @math with a grammar of size @math , uses @math bits and counts the number of occurrences of a pattern @math in time @math , for any constant @math . We also give the first index using @math bits, where @math is parsed by Lempel-Ziv into @math phrases, counting occurrences in time @math . The retrieval problem is the problem of associating data with keys in a set. Formally, the data structure must store a function f: U -> 0,1 ^r that has specified values on the elements of a given set S, a subset of U, |S|=n, but may have any value on elements outside S. Minimal perfect hashing makes it possible to avoid storing the set S, but this induces a space overhead of Theta(n) bits in addition to the nr bits needed for function values. In this paper we show how to eliminate this overhead. Moreover, we show that for any k query time O(k) can be achieved using space that is within a factor 1+e^ -k of optimal, asymptotically for large n. If we allow logarithmic evaluation time, the additive overhead can be reduced to O(log log n) bits whp. The time to construct the data structure is O(n), expected. A main technical ingredient is to utilize existing tight bounds on the probability of almost square random matrices with rows of low weight to have full row rank. In addition to direct constructions, we point out a close connection between retrieval structures and hash tables where keys are stored in an array and some kind of probing scheme is used. Further, we propose a general reduction that transfers the results on retrieval into analogous results on approximate membership, a problem traditionally addressed using Bloom filters. Again, we show how to eliminate the space overhead present in previously known methods, and get arbitrarily close to the lower bound. The evaluation procedures of our data structures are extremely simple (similar to a Bloom filter). For the results stated above we assume free access to fully random hash functions. However, we show how to justify this assumption using extra space o(n) to simulate full randomness on a RAM. Abstract One of the most important problems in information retrieval is determining the order of documents in the answer returned to the user. Many methods and algorithms for document ordering have been proposed. The method introduced in this paper differs from them especially in that it uses a probabilistic model of a document set. In this model documents are regarded as states of a Markov chain, where transition probabilities are directly proportional to similarities between documents. Steady-state probabilities reflect similarities of particular documents to the whole answer set. If documents are ordered according to these probabilities, at the top of a list there will be documents that are the best representatives of the set, and at the bottom those which are the worst representatives. The method was tested against databases INSPEC and Networked Computer Science Technical Reference Library (NCSTRL). Test results are positive. Values of the Kendall rank correlation coefficient indicate high similarity between rankings generated by the proposed method and rankings produced by experts. Results are comparable with rankings generated by the vector model using standard weighting schema tf·idf.