In addition to the frequency of terms in a document collection, the distribution of terms plays an important role in determining the relevance of documents. In this paper, a new approach for representing term positions in documents is presented. The approach allows an efficient evaluation of term-positional information at query evaluation time. Three applications are investigated: a function-based ranking optimization representing a user-defined document region, a query expansion technique based on overlapping the term distributions in the top-ranked documents, and cluster analysis of terms in documents. Experimental results demonstrate the effectiveness of the proposed approach.
One of the first approaches using abstract representations of term distributions in documents is Fourier Domain Scoring (FDS), proposed by @cite_8 . FDS performs a separate magnitude and phase analysis of term position signals to produce an optimized ranking. It creates an index based on page segmentation, storing term frequency and approximated positions in the document. FDS processes the indexed data using the to perform the corresponding spectral analysis.
@cite_8
2105660794
In addition to the frequency of terms in a document collection, the distribution of terms plays an important role in determining the relevance of documents for a given search query. In this paper, term distribution analysis using Fourier series expansion as a novel approach for calculating an abstract representation of term positions in a document corpus is introduced. Based on this approach, two methods for improving the evaluation of document relevance are proposed: (a) a function-based ranking optimization representing a user defined document region, and (b) a query expansion technique based on overlapping the term distributions in the top-ranked documents. Experimental results demonstrate the effectiveness of the proposed approach in providing new possibilities for optimizing the retrieval process. Shape is one of the primary low level image features in Content Based Image Retrieval (CBIR). Many shape representations and retrieval methods exist. However, most of those methods either do not well capture shape features or are difficult to do normalization (making matching difficult). Among them, methods based Fourier descriptors (FDs) achieve both good representation (perceptually meaningful) and easy normalization. Besides, FDs are easy to derive and compact in terms of representation. Design of FDs focuses on how to derive Fourier invariants from Fourier coefficients and how to obtain Fourier coefficients from shape signatures. Different Fourier invariants and shape signatures have been exploited to derive FDs. In this paper, we study different FDs and build a Java retrieval framework to compare shape retrieval performance using different FDs in terms of computation complexity, robustness, convergence speed and retrieval performance. The retrieval performance of the different FDs is compared using a standard shape database. Traditional interactive information retrieval systems function by creating inverted lists, or term indexes. For every term in the vocabulary, a list is created that contains the documents in which that term occurs and its relative frequency within each document. Retrieval algorithms then use these term frequencies alongside other collection statistics to identify the matching documents for a query. In this paper, we turn the process around: instead of indexing documents, we index query result sets. First, queries are run through a chosen retrieval system. For each query, the resulting document IDs are treated as terms and the score or rank of the document is used as the frequency statistic. An index of documents retrieved by basis queries is created. We call this index a reverted index. With reverted indexes, standard retrieval algorithms can retrieve the matching queries (as results) for a set of documents (used as queries). These recovered queries can then be used to identify additional documents, or to aid the user in query formulation, selection, and feedback. Current document retrieval methods use a vector space similarity measure to give scores of relevance to documents when related to a specific query. The central problem with these methods is that they neglect any spatial information within the documents in question. We present a new method, called Fourier Domain Scoring (FDS), which takes advantage of this spatial information, via the Fourier transform, to give a more accurate ordering of relevance to a document set. We show that FDS gives an improvement in precision over the vector space similarity measures for the common case of Web like queries, and it gives similar results to the vector space measures for longer queries. Large flash disks, or solid state drives (SSDs), have become an attractive alternative to magnetic hard disks, due to their high random read performance, low energy consumption and other features. However, writes, especially small random writes, on flash disks are inherently much slower than reads because of the erase-before-write mechanism. To address this asymmetry of read-write speeds in tree indexing on the flash disk, we propose FD-tree, a tree index designed with the logarithmic method and fractional cascading techniques. With the logarithmic method, an FD-tree consists of the head tree -- a small B+-tree on the top, and a few levels of sorted runs of increasing sizes at the bottom. This design is write-optimized for the flash disk; in particular, an index search will potentially go through more levels or visit more nodes, but random writes are limited to a small area -- the head tree, and are subsequently transformed into sequential ones through merging into the lower runs. With the fractional cascading technique, we store pointers, called fences, in lower level runs to speed up the search. Given an FD-tree of n entries, we analytically show that it performs an update in O(logB n) sequential I Os and completes a search in O(logB n) random I Os, where B is the flash page size. We evaluate FD-tree in comparison with representative B+-tree variants under a variety of workloads on three commodity flash SSDs. Our results show that FD-tree has a similar search performance to the standard B+-tree, and a similar update performance to the write-optimized B+-tree variant. As a result, FD-tree dominates the other B+-tree index variants on the overall performance on flash disks as well as on magnetic disks. Archiving official written documents such as invoices, reminders and account statements in business and private area gets more and more important. Creating appropriate index entries for document archives like sender's name, creation date or document number is a tedious manual work. We present a novel approach to handle automatic indexing of documents based on generic positional extraction of index terms. For this purpose we apply the knowledge of document templates stored in a common full text search index to find index positions that were successfully extracted in the past. The potential of doubling the spectrum efficiency of FD transmission motivates us to investigate FD-mmWave communication. To realize FD transmission in the mmWave band, we first introduce possible antenna configurations for FD-mmWave transmission. It is shown that, different from the cases in microwave band FD communications, the configuration with separate Tx Rx antenna arrays appears more flexible in SI suppression while it may increase some cost and area versus that with the same array. We then model the mmWave SI channel with separate Tx Rx arrays, where a near-field propagation model is adopted for the LOS path, and it is found that the established LOS-SI channel with separate Tx Rx arrays also shows spatial sparsity. Based on the SI channel, we further explore approaches to mitigate SI by signal processing, and we focus on a new cancellation approach in FD-mmWave communication, that is, beamforming cancellation. Centered on the CA constraint of the beamforming vectors, we propose several candidate solutions. Lastly, we consider an FD-mmWave multi-user scenario, and show that even if there are no FD users in an FD-mmWave cellular system, the FD benefit can still be exploited in the FD base station. Candidate solutions are also discussed to mitigate both SI and MUI simultaneously. The performance analysis of optimal decode-and-forward (DF) relay selection for the full-duplex (FD) mode in underlay cognitive radio (CR) networks is studied, with the impact of critical parameters such as the residual self-interference, the distributions of the received signal-to-noise ratio (SNR), and the outage probability, being analyzed. Different from the conventional relay selection schemes with half-duplex (HD) mode, the FD relay suffers from the impact of residual self-interference due to its simultaneous transmission and reception using the same channel. Furthermore, the exact closed-form expressions for the outage probabilities of the proposed FD relay-selection scheme over both independent Nakagami- @math and Rayleigh fading channels are derived, with the validity of the proposed analysis being proven by using simulation. Numerical results show that, in the presence of multiple candidate relays, an optimum relay-selection solution in terms of outage probability can always be achievable within the SNR range of (10 dB, 15 dB). Abstract Documents in computer-readable form can be used to provide information about other documents, i.e. those they cite. To do this efficiently requires procedures for computer recognition of citing statements. This is not easy, especially for multi-sentence citing statements. Computer recognition procedures have been developed which are accurate to the following extent: 73 of the words in statements selected by computer procedures as being citing statements are words which are correctly attributable to the corresponding documents. The retrieval effectiveness of computer-recognized citing statements was tested in the following way. First, for eight retrieval requests in inorganiic chemistry, average recall by search of Chemical Abstracts Service indexing and Chemical Abstracts abstract text words was found to be 50 . Words from citing statements referring to the papers to be retrieved were then added to the index terms and abstract words as additional access points, and searching was repeated. Average recall increased to 70 . Only words from citing statements published within a year of the cited papers were used. The retrieval effect of citing statement words alone (published within a year) without index or abstract terms was the following: average recall was 40 . When just the words of the titles of the cited papers were added to those citing statement words, average recall increased to 50 . Flat Datacenter Storage (FDS) is a high-performance, fault-tolerant, large-scale, locality-oblivious blob store. Using a novel combination of full bisection bandwidth networks, data and metadata striping, and flow control, FDS multiplexes an application's large-scale I O across the available throughput and latency budget of every disk in a cluster. FDS therefore makes many optimizations around data locality unnecessary. Disks also communicate with each other at their full bandwidth, making recovery from disk failures extremely fast. FDS is designed for datacenter scale, fully distributing metadata operations that might otherwise become a bottleneck. FDS applications achieve single-process read and write performance of more than 2GB s. We measure recovery of 92GB data lost to disk failure in 6.2 s and recovery from a total machine failure with 655GB of data in 33.7 s. Application performance is also high: we describe our FDS-based sort application which set the 2012 world record for disk-to-disk sorting. We investigate the open problem of characterizing the multiplexing gain offered by FD in a network of M cells (compared to the gain of two available on a single link). While self-interference cancellation is fundamental in realizing full duplex (FD) capability, the more challenging problem in a network-wide deployment of FD communication is a new form of uplink-downlink interference, namely UDI, caused by transmission of uplink clients on the downlink reception of other clients operating in the same frequency band during FD. We leverage spatial interference alignment (IA) as an effective approach to address UDI and characterize the scalability of the FD's multiplexing gain (in terms of M) by providing a closed-form expression. To the best of our knowledge, this is the first characterization of FD's multiplexing gain in a multi-cell network. We also provide an IA construction that can achieve the best scaling possible. Further, we extend our results to practical settings with limited number of clients and limited information sharing between access points. Many web documents are dynamic, with content changing in varying amounts at varying frequencies. However, current document search algorithms have a static view of the document content, with only a single version of the document in the index at any point in time. In this paper, we present the first published analysis of using the temporal dynamics of document content to improve relevance ranking. We show that there is a strong relationship between the amount and frequency of content change and relevance. We develop a novel probabilistic document ranking algorithm that allows differential weighting of terms based on their temporal characteristics. By leveraging such content dynamics we show significant performance improvements for navigational queries. Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent semantic space. In traditional information retrieval models, on the other hand, terms have discrete or local representations, and the relevance of a document is determined by the exact matches of query terms in the body text. We hypothesize that matching with distributed representations complements matching with traditional local representations, and that a combination of the two is favourable. We propose a novel document ranking model composed of two separate deep neural networks, one that matches the query and the document using a local representation, and another that matches the query and the document using learned distributed representations. The two networks are jointly trained as part of a single neural network. We show that this combination or 'duet' performs significantly better than either neural network individually on a Web page ranking task, and significantly outperforms traditional baselines and other recently proposed models based on neural networks. Conditional functional dependencies (CFDs) have recently been proposed as a useful integrity constraint to summarize data semantics and identify data inconsistencies. A CFD augments a functional dependency (FD) with a pattern tableau that defines the context (i.e., the subset of tuples) in which the underlying FD holds. While many aspects of CFDs have been studied, including static analysis and detecting and repairing violations, there has not been prior work on generating pattern tableaux, which is critical to realize the full potential of CFDs. This paper is the first to formally characterize a "good" pattern tableau, based on naturally desirable properties of support, confidence and parsimony. We show that the problem of generating an optimal tableau for a given FD is NP-complete but can be approximated in polynomial time via a greedy algorithm. For large data sets, we propose an "on-demand" algorithm providing the same approximation bound, that outperforms the basic greedy algorithm in running time by an order of magnitude. For ordered attributes, we propose the range tableau as a generalization of a pattern tableau, which can achieve even more parsimony. The effectiveness and efficiency of our techniques are experimentally demonstrated on real data. Patent prior art search is a type of search in the patent domain where documents are searched for that describe the work previously carried out related to a patent application. The goal of this search is to check whether the idea in the patent application is novel. Vocabulary mismatch is one of the main problems of patent retrieval which results in low retrievability of similar documents for a given patent application. In this paper we show how the term distribution of the cited documents in an initially retrieved ranked list can be used to address the vocabulary mismatch. We propose a method for query modeling estimation which utilizes the citation links in a pseudo relevance feedback set. We first build a topic dependent citation graph, starting from the initially retrieved set of feedback documents and utilizing citation links of feedback documents to expand the set. We identify the important documents in the topic dependent citation graph using a citation analysis measure. We then use the term distribution of the documents in the citation graph to estimate a query model by identifying the distinguishing terms and their respective weights. We then use these terms to expand our original query. We use CLEF-IP 2011 collection to evaluate the effectiveness of our query modeling approach for prior art search. We also study the influence of different parameters on the performance of the proposed method. The experimental results demonstrate that the proposed approach significantly improves the recall over a state-of-the-art baseline which uses the link-based structure of the citation graph but not the term distribution of the cited documents. In this paper we propose a segmentation-free query by string word spotting method. Both the documents and query strings are encoded using a recently proposed word representation that projects images and strings into a common attribute space based on a Pyramidal Histogram of Characters (PHOC). These attribute models are learned using linear SVMs over the Fisher Vector [8] representation of the images along with the PHOC labels of the corresponding strings. In order to search through the whole page, document regions are indexed per character bi-gram using a similar attribute representation. On top of that, we propose an integral image representation of the document using a simplified version of the attribute model for efficient computation. Finally we introduce a re-ranking step in order to boost retrieval performance. We show state-of-the-art results for segmentation-free query by string word spotting in single-writer and multi-writer standard datasets. Versioned document collections are collections that contain multiple versions of each document. Important examples are Web archives, Wikipedia and other wikis, or source code and documents maintained in revision control systems. Versioned document collections can become very large, due to the need to retain past versions, but there is also a lot of redundancy between versions that can be exploited. Thus, versioned document collections are usually stored using special differential (delta) compression techniques, and a number of researchers have recently studied how to exploit this redundancy to obtain more succinct full-text index structures. In this paper, we study index organization and compression techniques for such versioned full-text index structures. In particular, we focus on the case of positional index structures, while most previous work has focused on the non-positional case. Building on earlier work in [zs:redun], we propose a framework for indexing and querying in versioned document collections that integrates non-positional and positional indexes to enable fast top-k query processing. Within this framework, we define and study the problem of minimizing positional index size through optimal substring partitioning. Experiments on Wikipedia and web archive data show that our techniques achieve significant reductions in index size over previous work while supporting very fast query processing. Vague information is common in many database applications due to intensive data dissemination arising from different pervasive computing sources, such as the high volume data obtained from sensor networks and mobile communications. In this paper, we utilize functional dependencies (FDs) and inclusion dependencies (INDs), which are the most fundamental integrity constraints that arise in practice in relational databases, to maintain the consistency of a vague database. First, we tackle the problem, given a vague relation r and a set of FDs F, of how to obtain the ''best'' approximation of r with respect to F when taking into account the median membership (m) and the imprecision membership (i) thresholds. Using these two thresholds of a vague set, we define the notion of mi-overlap between vague sets and a merge operation on r. Second, we consider, given a vague database d and a set of INDs N, how to obtain the minimal possible change in value-precision for d. Satisfaction of an FD in r is defined in terms of values being mi-overlapping while satisfaction of an IND in d is defined in terms of value-precision. We show that Lien's and Atzeni's axiom system is sound and complete for FDs being satisfied in vague relations and that 's axiom system is sound and complete for INDs being satisfied in vague databases. Finally, we study the chase procedure VChase(d,[email protected]?N) as a means to maintain consistency of d with respect to F and N. Our main result is that the output of the procedure is the most object-precise approximation of r with respect to F and the minimum value-precision change of d with respect to N. The complexity of VChase(r,F) is polynomial time in the sizes of r and F whereas the complexity of VChase(d,[email protected]?N) is exponential. A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs. We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features. Scholars are increasingly citing electronic “web references” which are not preserved in libraries or full text archives. WebCite is a new standard for citing web references. To “webcite” a document involves archiving the cited Web page through www.webcitation.org and citing the WebCite permalink instead of (or in addition to) the unstable live Web page. This journal has amended its “instructions for authors” accordingly, asking authors to archive cited Web pages before submitting a manuscript. Almost 200 other journals are already using the system. We discuss the rationale for WebCite, its technology, and how scholars, editors, and publishers can benefit from the service. Citing scholars initiate an archiving process of all cited Web references, ideally before they submit a manuscript. Authors of online documents and websites which are expected to be cited by others can ensure that their work is permanently available by creating an archived copy using WebCite and providing the citation information including the WebCite link on their Web document(s). Editors should ask their authors to cache all cited Web addresses (Uniform Resource Locators, or URLs) “prospectively” before submitting their manuscripts to their journal. Editors and publishers should also instruct their copyeditors to cache cited Web material if the author has not done so already. Finally, WebCite can process publisher submitted “citing articles” (submitted for example as eXtensible Markup Language [XML] documents) to automatically archive all cited Web pages shortly before or on publication. Finally, WebCite can act as a focussed crawler, caching retrospectively references of already published articles. Copyright issues are addressed by honouring respective Internet standards (robot exclusion files, no-cache and no-archive tags). Long-term preservation is ensured by agreements with libraries and digital preservation organizations. The resulting WebCite Index may also have applications for research assessment exercises, being able to measure the impact of Web services and published Web documents through access and Web citation metrics. @PARASPLIT [J Med Internet Res 2005;7(5):e60]