5516407
Condensed cube: an effective approach to reducing data cube size
Pre-computed data cube facilitates OLAP (on-line analytical processing). It is well-known that data cube computation is an expensive operation. While most algorithms have been devoted to optimizing memory management and reducing computation costs, less work has addressed a fundamental issue: the size of a data cube is huge when a large base relation with a large number of attributes is involved. In this paper, we propose a new concept, called a condensed data cube. The condensed cube is of much smaller size than a complete non-condensed cube. More importantly, it is a fully pre-computed cube without compression, and, hence, it requires neither decompression nor further aggregation when answering queries. Several algorithms for computing a condensed cube are proposed. Results of experiments on the effectiveness of condensed data cube are presented, using both synthetic and real-world data. The results indicate that the proposed condensed cube can reduce both the cube size and therefore its computation time.
6717365
Efficient approximate entity extraction with edit distance constraints
Named entity recognition aims at extracting named entities from unstructured text. A recent trend of named entity recognition is finding approximate matches in the text with respect to a large dictionary of known entities, as the domain knowledge encoded in the dictionary helps to improve the extraction performance.
 In this paper, we study the problem of approximate dictionary matching with edit distance constraints. Compared to existing studies using token-based similarity constraints, our problem definition enables us to capture typographical or orthographical errors, both of which are common in entity extraction tasks yet may be missed by token-based similarity constraints. Our problem is technically challenging as existing approaches based on q-gram filtering have poor performance due to the existence of many short entities in the dictionary. Our proposed solution is based on an improved neighborhood generation method employing novel partitioning and prefix pruning techniques. We also propose an efficient document processing algorithm that minimizes unnecessary comparisons and enumerations and hence achieves good scalability. We have conducted extensive experiments on several publicly available named entity recognition datasets. The proposed algorithm outperforms alternative approaches by up to an order of magnitude.
3430950
Efficient Processing of XML Path Queries Using the Disk-based F&B Index
With the proliferation of XML data and applications on the Internet, efficient XML query processing techniques are in great demand. Answering queries using XML indexes is a natural approach. A number of XML indexes have been proposed in the literature: among them, FB the result is a disk-based F&B Index with good clustering properties. In addition, novel query processing algorithms exploiting the physical organization of the disk-based F&B Indexes are proposed. Experimental results verify that our disk-based F&B Index can scale up for large data size with good query performance compared with state-of-the-art XML query processing algorithms.
10066890
Quantile-based KNN over multi-valued objects
K Nearest Neighbor search has many applications including data mining, multi-media, image processing, and monitoring moving objects. In this paper, we study the problem of KNN over multi-valued objects. We aim to provide effective and efficient techniques to identify KNN sensitive to relative distributions of objects.We propose to use quantiles to summarize relative-distribution-sensitive K nearest neighbors. Given a query Q and a quantile φ ∈ (0, 1], we firstly study the problem of efficiently computing K nearest objects based on a φ-quantile distance (e.g. median distance) from each object to Q. The second problem is to retrieve the K nearest objects to Q based on overall distances in the “best population” (with a given size specified by φ-quantile) for each object. While the first problem can be solved in polynomial time, we show that the 2nd problem is NP-hard. A set of efficient, novel algorithms have been proposed to give an exact solution for the first problem and an approximate solution for the second problem with the approximation ratio 2. Extensive experiment demonstrates that our techniques are very efficient and effective.
2285434
Towards multidimensional subspace skyline analysis
The skyline operator is important for multicriteria decision-making applications. Although many recent studies developed efficient methods to compute skyline objects in a given space, none of them considers skylines in multiple subspaces simultaneously. More importantly, the fundamental problem on the semantics of skylines remains open: Why and in which subspaces is (or is not) an object in the skyline? Practically, users may also be interested in the skylines in any subspaces. Then, what is the relationship between the skylines in the subspaces and those in the super-spaces? How can we effectively analyze the subspace skylines? Can we efficiently compute skylines in various subspaces and answer various analytical queries?In this article, we tackle the problem of multidimensional subspace skyline computation and analysis. We explore skylines in subspaces. First, we propose the concept of Skycube, which consists of skylines of all possible nonempty subspaces of a given full space. Once a Skycube is materialized, any subspace skyline queries can be answered online. However, Skycube cannot fully address the semantic concerns and may contain redundant information. To tackle the problem, we introduce a novel notion of skyline group which essentially is a group of objects that coincide in the skylines of some subspaces. We identify the decisive subspaces that qualify skyline groups in the subspace skylines. The new notions concisely capture the semantics and the structures of skylines in various subspaces. Multidimensional roll-up and drill-down analysis is introduced. We also develop efficient algorithms to compute Skycube, skyline groups and their decisive subspaces. A systematic performance study using both real data sets and synthetic data sets is reported to evaluate our approach.
8670128
Top-k Set Similarity Joins
Similarity join is a useful primitive operation underlying many applications, such as near duplicate Web page detection, data integration, and pattern recognition. Traditional similarity joins require a user to specify a similarity threshold. In this paper, we study a variant of the similarity join, termed top-k set similarity join. It returns the top-k pairs of records ranked by their similarities, thus eliminating the guess work users have to perform when the similarity threshold is unknown before hand. An algorithm, topk-join, is proposed to answer top-k similarity join efficiently. It is based on the prefix filtering principle and employs tight upper bounding of similarity values of unseen pairs. Experimental results demonstrate the efficiency of the proposed algorithm on large-scale real datasets.
13221631
Fast intrusion detection based on a non-negative matrix factorization model
In this paper, we present an efficient fast anomaly intrusion detection model incorporating a large amount of data from various data sources. A novel method based on non-negative matrix factorization (NMF) is presented to profile program and user behaviors of a computer system. A large amount of high-dimensional data is collected in our experiments and divided into smaller data blocks by a specific scheme. The system call data is divided into blocks by processes, while command data is divided into consecutive blocks with a fixed length. The frequencies of individual elements in each block of data are computed and placed column by column as data vectors to construct a matrix representation. NMF is employed to reduce the high-dimensional data vectors and anomaly detection can be realized as a very simple classifier in low dimensions. Experimental results show that the model presented in this paper is promising in terms of detection accuracy, computation efficiency and implementation for fast intrusion detection.
13632117
A Novel Intrusion Detection Method Based on Principle Component Analysis in Computer Security
Intrusion detection is an important technique in the defense-in-depth network security framework and a hot topic in computer security in recent years. In this paper, a new intrusion detection method based on Principle Component Analysis (PCA) with low overhead and high efficiency is presented. System call data and command sequences data are used as information sources to validate the proposed method. The frequencies of individual system calls in a trace and individual commands in a data block are computed and then data column vectors which represent the traces and blocks of the data are formed as data input. PCA is applied to reduce the high dimensional data vectors and distance between a vector and its projection onto the subspace reduced is used for anomaly detection. Experimental results show that the proposed method is promising in terms of detection accuracy, computational expense and implementation for real-time intrusion detection.
2197213
Reconstruction of ancestral gene order after segmental duplication and gene loss
As gene order evolves through a variety of chromosomal rearrangements, conserved segments provide important insight into evolutionary relationships and functional roles of genes. However, gene loss within otherwise conserved segments, as typically occurs following large-scale genome duplication, has received limited algorithmic study. This has been a major impediment to comparative genomics in certain taxa, such as plants and fish. We propose a heuristic algorithm/or the inference of ancestral gene order in a set of related genomes that have undergone large-scale duplication and gene loss. First, approximately conserved (i.e. homologous) segments are identified using pairwise local genome alignment. Second, homologous segments are iteratively clustered under the control of two parameters, (1) the minimal required number of shared genes between two clusters and (2) the maximal allowed number of rearrangement breakpoints along the lineage leading to each descendant segment. Finally, we compute an estimated ancestral gene order for each cluster that is optimal in some sense. We evaluate the performance of this algorithm on simulated data that models a genome evolving by large-scale duplication, duplicate gene loss, transposition, translocation, and inversion. The results suggest that long segments of ancestral gene order may be reconstructed following moderate levels of rearrangement with only minor loss of accuracy.
26642486
Computing Repairs for Inconsistent XML Document Using Chase
An XML document is inconsistent if it violates predefined integrity constraints. In this paper, we consider how to compute repairs for an inconsistent XML document. Here repair is defined as the data consistent with the integrity constraints, and also minimally differs from the original document. Based on a repair framework by introducing a chase method, in this paper, we discuss the repairs computing problem and implement a prototype. First we discuss some key points about mends generation and repairs chasing. Next we give a cost model for this repair framework, which can be used to evaluate the cost of each repair. Finally we implement prototypes of our method, and evaluate our framework and algorithms in the experiment.
14948142
ADenTS: An Adaptive Density-Based Tree Structure for Approximating Aggregate Queries over Real Attributes
In many fields and applications, it is critical for users to make decisions through OLAP queries. How to promote accuracy and efficiency while answering multiple aggregate queries, e.g. COUNT, SUM, AVG, MAX, MIN and MEDIAN? It has been the urgent problem in the fields of OLAP and data summarization recently. There have been a few solutions such as MRA-Tree and GENHIST for it. However, they could only answer a certain aggregate query which was defined in a particular data cube with some limited applications. In this paper, we develop a novel framework ADenTS, i.e. Adaptive Density-based Tree Structure, to answer various types of aggregate queries within a single data cube. We represent the whole cube by building a coherent tree structure. Several techniques for approximation are also proposed. The experimental results show that our method outperforms others in effectiveness.