765245
CARE: Finding Local Linear Correlations in High Dimensional Data
Finding latent patterns in high dimensional data is an important research problem with numerous applications. Existing approaches can be summarized into 3 categories: feature selection, feature transformation (or feature projection) and projected clustering. Being widely used in many applications, these methods aim to capture global patterns and are typically performed in the full feature space. In many emerging biomedical applications, however, scientists are interested in the local latent patterns held by feature subsets, which may be invisible via any global transformation. In this paper, we investigate the problem of finding local linear correlations in high dimensional data. Our goal is to find the latent pattern structures that may exist only in some subspaces. We formalize this problem as finding strongly correlated feature subsets which are supported by a large portion of the data points. Due to the combinatorial nature of the problem and lack of monotonicity of the correlation measurement, it is prohibitively expensive to exhaustively explore the whole search space. In our algorithm, CARE, we utilize spectrum properties and effective heuristic to prune the search space. Extensive experimental results show that our approach is effective in finding local linear correlations that may not be identified by existing methods.
12021051
COE: A General Approach for Efficient Genome-Wide Two-Locus Epistasis Test in Disease Association Study
The availability of high density single nucleotide polymorphisms (SNPs) data has made genome-wide association study computationally challenging. Two-locus epistasis (gene-gene interaction) detection has attracted great research interest as a promising method for genetic analysis of complex diseases. In this paper, we propose a general approach, COE, for efficient large scale gene-gene interaction analysis, which supports a wide range of tests. In particular, we show that many commonly used statistics are convex functions. From the observed values of the events in two-locus association test, we can develop an upper bound of the test value. Such an upper bound only depends on single-locus test and the genotype of the SNP-pair. We thus group and index SNP-pairs by their genotypes. This indexing structure can benefit the computation of all convex statistics. Utilizing the upper bound and the indexing structure, we can prune most of the SNP-pairs without compromising the optimality of the result. Our approach is especially efficient for large permutation test. Extensive experiments demonstrate that our approach provides orders of magnitude performance improvement over the brute force approach.
9042204
CRD: fast co-clustering on large datasets utilizing sampling-based matrix decomposition
The problem of simultaneously clustering columns and rows (co-clustering) arises in important applications, such as text data mining, microarray analysis, and recommendation system analysis. Compared with the classical clustering algorithms, co-clustering algorithms have been shown to be more effective in discovering hidden clustering structures in the data matrix. The complexity of previous co-clustering algorithms is usually O(m X n), where m and n are the numbers of rows and columns in the data matrix respectively. This limits their applicability to data matrices involving a large number of columns and rows. Moreover, some huge datasets can not be entirely held in main memory during co-clustering which violates the assumption made by the previous algorithms. In this paper, we propose a general framework for fast co-clustering large datasets, CRD. By utilizing recently developed sampling-based matrix decomposition methods, CRD achieves an execution time linear in m and n. Also, CRD does not require the whole data matrix be in the main memory. We conducted extensive experiments on both real and synthetic data. Compared with previous co-clustering algorithms, CRD achieves competitive accuracy but with much less computational cost.
16446501
Mining frequent closed patterns in microarray data
Microarray data typically contains a large number of columns and a small number of rows, which poses a great challenge for existing frequent (closed) pattern mining algorithms that discover patterns in item enumeration space. In this paper, we propose two algorithms that explore the row enumeration space to mine frequent closed patterns. Several experiments on real-life gene expression data show that the algorithms are faster than existing algorithms, including CLOSET, CHARM, CLOSET+ and CARPENTER.
268487
Mining non-redundant high order correlations in binary data
Many approaches have been proposed to find correlations in binary data. Usually, these methods focus on pair-wise correlations. In biology applications, it is important to find correlations that involve more than just two features. Moreover, a set of strongly correlated features should be non-redundant in the sense that the correlation is strong only when all the interacting features are considered together. Removing any feature will greatly reduce the correlation.In this paper, we explore the problem of finding non-redundant high order correlations in binary data. The high order correlations are formalized using multi-information, a generalization of pairwise mutual information. To reduce the redundancy, we require any subset of a strongly correlated feature subset to be weakly correlated. Such feature subsets are referred to as Non-redundant Interacting Feature Subsets (NIFS). Finding all NIFSs is computationally challenging, because in addition to enumerating feature combinations, we also need to check all their subsets for redundancy. We study several properties of NIFSs and show that these properties are useful in developing efficient algorithms. We further develop two sets of upper and lower bounds on the correlations, which can be incorporated in the algorithm to prune the search space. A simple and effective pruning strategy based on pair-wise mutual information is also developed to further prune the search space. The efficiency and effectiveness of our approach are demonstrated through extensive experiments on synthetic and real-life datasets.
7252877
FARMER: finding interesting rule groups in microarray datasets
Microarray datasets typically contain large number of columns but small number of rows. Association rules have been proved to be useful in analyzing such datasets. However, most existing association rule mining algorithms are unable to efficiently handle datasets with large number of columns. Moreover, the number of association rules generated from such datasets is enormous due to the large number of possible column combinations.In this paper, we describe a new algorithm called FARMER that is specially designed to discover association rules from microarray datasets. Instead of finding individual association rules, FARMER finds interesting rule groups which are essentially a set of rules that are generated from the same set of rows. Unlike conventional rule mining algorithms, FARMER searches for interesting rules in the row enumeration space and exploits all user-specified constraints including minimum support, confidence and chi-square to support efficient pruning. Several experiments on real bioinformatics datasets show that FARMER is orders of magnitude faster than previous association rule mining algorithms.
9260921
Complexity adaptive quantization for intra-frames in very low bit rate video coding
Conventional rate control schemes focus on the problem of finding an optimal quantization value for P- and B-frames. No rate control is available for the encoding of I-frames. This could pose big problems due to the large number of bits an I-frame could generate, and due to the fact that the number of bits varies drastically from sequence to sequence, depending on their image complexity. This problem becomes severe especially at very low bit rate. Therefore, a mechanism to allocate data bits to an I-frame according to its complexity is indispensable in order to have constant coding quality. This work presents a mechanism to establish for I-frames a generic relationship between the quantization value, data bits, and their image complexity. Experimental results show that this generic relationship provides a fairly accurate estimation of quantization value for an I-frame at given data bits and image complexity, and is very useful in controlling the data bits generated by an I-frame.
6178919
Proactive frame-skipping decision scheme for variable frame rate video coding
Many rate control algorithms focus on the adjustment of quantisation values to retain a certain buffer level, and arbitrary frame-skipping is often needed to keep the buffer from overflow at very low bit-rates. A content adaptive rate control is proposed to optimise the balance between spatial and temporal quality via active frame-skipping. The occurrence of frame-skipping is jointly dependent on the temporal and spatial quality of the video, and on the fullness of the buffer. This helps to achieve a consistent spatial and temporal quality and to enhance the overall perceptual quality. Experimental results show that the new scheme is simple but very effective, with large average PSNR gains and consistently improved visual quality, and the improvement in perceptual quality is much more significant than that in average PSNRs.
10609537
A new set of moment invariants for handwritten numeral recognition
In this paper, a new set of aspect invariant moments for handwritten numeral recognition are presented. These new moments exhibit two useful properties. Firstly, they are aspect invariant. This eliminates the need for size normalization of the unconstrained numerals. Secondly, their dynamic range remains constant with moment order. This overcomes the problem of diminishing high order moments, which occurs when other moment invariants are used. Thus, aspect invariant moments are particularly suitable for use with neural networks. Experimental results (using a multilayer perceptron and the backpropagation learning rule) show that a very high recognition rate (98.73%) and low substitution rate (1.06%) can be achieved on a totally unconstrained handwritten numeral database.<<ETX>>
16926436
Optimal Interdiction of Unreactive Markovian Evaders
The interdiction problem arises in a variety of areas including military logistics, infectious disease control, and counter-terrorism. In the typical formulation of network interdiction, the task of the interdictor is to find a set of edges in a weighted network such that the removal of those edges would maximally increase the cost to an evader of traveling on a path through the network. 
 
Our work is motivated by cases in which the evader has incomplete information about the network or lacks planning time or computational power, e.g. when authorities set up roadblocks to catch bank robbers, the criminals do not know all the roadblock locations or the best path to use for their escape. 
 
We introduce a model of network interdiction in which the motion of one or more evaders is described by Markov processes and the evaders are assumed not to react to interdiction decisions. The interdiction objective is to find an edge set of size B , that maximizes the probability of capturing the evaders. 
 
We prove that similar to the standard least-cost formulation for deterministic motion this interdiction problem is also NP-hard. But unlike that problem our interdiction problem is submodular and the optimal solution can be approximated within 1 *** 1/e using a greedy algorithm. Additionally, we exploit submodularity through a priority evaluation strategy that eliminates the linear complexity scaling in the number of network edges and speeds up the solution by orders of magnitude. Taken together the results bring closer the goal of finding realistic solutions to the interdiction problem on global-scale networks.
16039054
Feature selection for ranking using boosted trees
Modern search engines have to be fast to satisfy users, so there are hard back-end latency requirements. The set of features useful for search ranking functions, though, continues to grow, making feature computation a latency bottleneck. As a result, not all available features can be used for ranking, and in fact, much of the time, only a small percentage of these features can be used. Thus, it is crucial to have a feature selection mechanism that can find a subset of features that both meets latency requirements and achieves high relevance. To this end, we explore different feature selection methods using boosted regression trees, including both greedy approaches (selecting the features with highest relative importance as computed by boosted trees; discounting importance by feature similarity and a randomized approach. We evaluate and compare these approaches using data from a commercial search engine. The experimental results show that the proposed randomized feature selection with feature-importance-based backward elimination outperforms greedy approaches and achieves a comparable relevance with 30 features to a full-feature model trained with 419 features and the same modeling parameters.