Abstract We consider document listing on string collections, that is, finding in which strings a given pattern appears. In particular, we focus on repetitive collections: a collection of size N over alphabet [ 1 , σ ] is composed of D copies of a string of size n, and s edits are applied on ranges of copies. We introduce the first document listing index with size O ˜ ( n + s ) , precisely O ( ( n lg ⁡ σ + s lg 2 ⁡ N ) lg ⁡ D ) bits, and with useful worst-case time guarantees: Given a pattern of length m, the index reports the ndoc > 0 strings where it appears in time O ( m lg 1 + ϵ ⁡ N ⋅ ndoc ) , for any constant ϵ > 0 (and tells in time O ( m lg ⁡ N ) if ndoc = 0 ). Our technique is to augment a range data structure that is commonly used on grammar-based indexes, so that instead of retrieving all the pattern occurrences, it computes useful summaries on them. We show that the idea has independent interest: we introduce the first grammar-based index that, on a text T [ 1 , N ] with a grammar of size r, uses O ( r lg ⁡ N ) bits and counts the number of occurrences of a pattern P [ 1 , m ] in time O ( m 2 + m lg 2 + ϵ ⁡ r ) , for any constant ϵ > 0 . We also give the first index using O ( z lg ⁡ ( N z ) lg ⁡ N ) bits, where T is parsed by Lempel–Ziv into z phrases, counting occurrences in time O ( m lg 2 + ϵ ⁡ N ) .
The last previous work is PDL @cite_1 , which stores inverted lists at sampled nodes in the suffix tree of @math , and then grammar-compresses the set of inverted lists. For a sampling step @math , it requires @math bits plus the (unbounded) space of the inverted lists. Searches that lead to the sampled nodes have their answers precomputed, whereas the others cover a suffix array range of size @math and are solved by brute force in time @math .
@cite_1
2141957180
Given a set @math of @math strings of total length @math , our task is to report the "most relevant"strings for a given query pattern @math . This involves somewhat more advanced query functionality than the usual pattern matching, as some notion of "most relevant" is involved. In information retrieval literature, this task is best achieved by using inverted indexes. However, inverted indexes work only for some predefined set of patterns. In the pattern matching community, the most popular pattern-matching data structures are suffix trees and suffix arrays. However, a typical suffix tree search involves going through all the occurrences of the pattern over the entire string collection, which might be a lot more than the required relevant documents. The first formal framework to study such kind of retrieval problems was given by [Muthukrishnan, 2002]. He considered two metrics for relevance: frequency and proximity. He took a threshold-based approach on these metrics and gave data structures taking @math words of space. We study this problem in a slightly different framework of reporting the top @math most relevant documents (in sorted order) under similar and more general relevance metrics. Our framework gives linear space data structure with optimal query times for arbitrary score functions. As a corollary, it improves the space utilization for the problems in [Muthukrishnan, 2002] while maintaining optimal query performance. We also develop compressed variants of these data structures for several specific relevance metrics.