15440093
Pattern discovery: a data driven approach to decision support
Decision support nowadays is more and more targeted to large scale complicated systems and domains. The success of a decision support system relies mainly on its capability of processing large amounts of data and efficiently extracting useful knowledge from the data, especially knowledge which is previously unknown to the decision makers. With a large scale system, traditional knowledge acquisition models become inefficient and/or more biased, due to the subjectivity of the experts or the pre-assumptions of certain ideas or algorithmic procedures. Today, with the rapid development of computer technologies, the capability of collecting data has been greatly advanced. Data becomes the most valuable resource for an organization. We present a fundamental framework toward intelligent decision support by analyzing a large amount of mixed-mode data (data with a mixture of continuous and categorical values) in order to bridge the subjectivity and objectivity of a decision support process. By considering significant associations of artifacts (events) inherent in the data as patterns, we define patterns as statistically significant associations among feature values represented by joint events or hypercells in the feature space. We then present an algorithm which automatically discovers statistically significant hypercells (patterns) based on: 1) a residual analysis, which tests the significance of the deviation when the occurrence of a hypercell differs from its expectation, and 2) an optimization formulation to enable recursive discovery. By discovering patterns from data sets based on such an objective measure, the nature of the problem domain will be revealed. The patterns can then be applied to solve specific problems as being interpreted or inferred with.
10246554
An Overview of Associative Classifiers
Associative classification is a new classification approach integrating association mining and classification. It becomes a significant tool for knowledge discovery and data mining. This paper presents a critical overview of certain aspects on modelling associative classifiers including event association mining, computational complexity in mining associations, methods of constructing associative classifiers based on the association patterns discovered, some similarities and differences among the reported associative classification systems, and experimental comparisons of these associative classifiers with the conventional classification system such as C4.5.
22289409
Attribute Clustering for Grouping, Selection, and Classification of Gene Expression Data
12250963
Boosting for Learning Multiple Classes with Imbalanced Class Distribution
Classification of data with imbalanced class distribution has posed a significant drawback of the performance attainable by most standard classifier learning algorithms, which assume a relatively balanced class distribution and equal misclassification costs. This learning difficulty attracts a lot of research interests. Most efforts concentrate on bi-class problems. However, bi-class is not the only scenario where the class imbalance problem prevails. Reported solutions for bi-class applications are not applicable to multi-class problems. In this paper, we develop a cost-sensitive boosting algorithm to improve the classification performance of imbalanced data involving multiple classes. One barrier of applying the cost-sensitive boosting algorithm to the imbalanced data is that the cost matrix is often unavailable for a problem domain. To solve this problem, we apply Genetic Algorithm to search the optimum cost setup of each class. Empirical tests show that the proposed cost-sensitive boosting algorithm improves the classification performances of imbalanced data sets significantly.
6225988
A global optimal algorithm for class-dependent discretization of continuous data
This paper presents a new method to convert continuous variables into discrete variables for inductive machine learning. The method can be applied to pattern classification problems in machine learning and data mining. The discretization process is formulated as an optimization problem. We first use the normalized mutual information that measures the interdependence between the class labels and the variable to be discretized as the objective function, and then use fractional programming (iterative dynamic programming) to find its optimum. Unlike the majority of class-dependent discretization methods in the literature which only find the local optimum of the objective functions, the proposed method, OCDD, or Optimal Class-Dependent Discretization, finds the global optimum. The experimental results demonstrate that this algorithm is very effective in classification when coupled with popular learning systems such as C4.5 decision trees and Naive-Bayes classifier. It can be used to discretize continuous variables for many existing inductive learning systems.
344886
From Association to Classification: Inference Using Weight of Evidence
Association and classification are two important tasks in data mining and knowledge discovery. Intensive studies have been carried out in both areas. But, how to apply discovered event associations to classification is still seldom found in current publications. Trying to bridge this gap, this paper extends our previous paper on significant event association discovery to classification. We propose to use weight of evidence to evaluate the evidence of a significant event association in support of, or against, a certain class membership. Traditional weight of evidence in information theory is extended here to measure the event associations of different orders with respect to a certain class. After the discovery of significant event associations inherent in a data set, it is easy and efficient to apply the weight of evidence measure for classifying an observation according to any attribute. With this approach, we achieve flexible prediction.
12791479
Group-by Query Process in Middleware of Large Scale Data Intensive Systems
Large scale data intensive systems are available in many fields in recent years, and it’s a severe challenge for group-by query of large volume of data in a cluster based on shared-nothing architecture. This paper proposes a design of a parallel query engine (PQE) and its asynchronous improvement (APQE) for group-by queries. PQE and APQE support for pipelined query processing and develop maximum degree of pipeline parallelism. APQE further eliminates the synchronous overhead of multi nodes parallelism, and returns part of final result as early as possible if no data dependency exists. Experimental results demonstrate that, compared to previous 2-step query engine, PQE and APQE can make a significant performance improvement for group-by query of large data sets in a shared-nothing cluster system, as well as obviously better scalability.
8581173
Recent database challenges in China on data consolidation and integration
During the first phase of IT exploitation in China, a lot of isolated systems were developed and deployed over all the sites where the business took place. Each system typically dealt with one simple local business application. Since the mid 90's, data consolidation and data integration became critical to address the business need for cheaper cost, shorter time to the market as well as better business decision and control. Without consolidated and integrated data, there is no way to support new and complex applications for rapid and large scale business growth in China across lots of industries such as financial, telecom, government, manufacture and distribution.
 When database is used for companies and government agencies in China, the large Chinese population makes the database volume huge. A nation-wide bank or an insurance company has hundreds of millions accounts easily. A province-level telecom service provider has tens of millions subscribers. In certain domains, such as banking and telecom, many applications are executed as on-line transactions which ask for large transaction throughput with good response time. A province-level telecom service provider expects to handle up to 100 billion calls in a month. In this case, most of the phone calls will be pre-paid type and these calls need to be handled in near-real time.
 In order to satisfy the performance requirement, some applications implement application cache on application servers. This approach improves the application performance in getting data but it also introduces the complexity and longer delay when a failure in application occurs. Database management systems can provide client cache for powerful application servers in these days. As more and more application servers are added to respond large number of concurrent users and high volume transaction throughput, database management system must be highly scalable. Both logical and physical contentions should be avoided or greatly reduced. Continuous availability and high reliability are more critical for a consolidated large database because many users and applications rely on it. Both planned outage and unplanned outage should be avoided or minimized through appropriate duplication and redundancy. Smooth transition through cooperative versions of data helps reduce planned outages.
 Swift switch over to hot stand-by database system helps minimize the unplanned outages. Database management systems are expected to be autonomic and remain healthy in resource utilization with minimal administrative involvement. As a summary, data consolidation in China asks for low-cost, high performance database systems with high scalability, availability and reliability on very large database volume.
 Data integration in China often deals with both a horizontal layer of multiple lines of business and a vertical organizational structure. For government agencies, there are at least four levels across nation, province, district and county. There are a few industry specific data models and solutions to address the requirements of data integration. However, a common data integration platform has been constantly requested by many IT projects of large scale. In many cases, business control processes need to integrate business operation processes to support near-real time decision and control. There is a high demand for a practical and well-proven methodology with tool sets which can support data integration life cycle of architect, design, development, deployment and change management.
 Rapid economic and business growth in combination of the join to WTO created big demand in IT evolution across China business, industry and government. There are many demands and opportunities of data consolidation and integration with large data volume and system complexity. The demand of low-cost solutions makes it even more challenging.
882072
Scalable, reliable marshalling and organization of distributed large scale data onto enterprise storage environments
Emerging technologies in high speed NAS, hierarchical storage management systems, and networked systems that virtualize interconnected storage over IP and fiber-channel networks, promise to consolidate distributed data stores onto large-scale professionally managed enterprise storage environments. We describe the software architecture of the PAWN (producer-archive workflow network) environment that enables scalable, reliable marshalling and organization of distributed data into such enterprise storage environments. PAWN was initially developed to capture the core elements required for long term preservation of digital objects as identified by researchers in the digital library and archiving communities. In this paper, we show how PAWN can be extended to enable multiple clients at a number of distributed sites to prepare, organize, and bulk transfer large scale data onto clusters of servers that securely verify the integrity of the data, register the metadata, and store the data into an enterprise storage environment. PAWN allows detailed description, auditing, and organization of the data, and hence allows for efficient management, access, and disaster recovery. The basic software components are based on open standards and Web technologies, and hence are platform independent.
1683130
PLA-based runtime dynamism in support of privacy-enhanced Web personalization
Software product line architectures (PLAs) have been widely recognized as a successful approach in industrial software development for improving productivity, software quality and time-to-market. In this paper, we focus on the usage of a PLA for a quite different purpose, namely, handling privacy constraints in Web personalization. To provide personalized services such as customized recommendations, a personalized Website collects users' personal data, which raises various privacy concerns. We aim at reconciling the benefits of web personalization with privacy constraints that come from users themselves as well as from privacy legislations and regulations that apply to a given user. We propose a dynamic, privacy-enabling personalization infrastructure and conceive it as a PLA. This infrastructure allows for dynamically selecting and instantiating personalization architectures that provide personalized services to each individual user and comply with the prevailing privacy constraints
8929565
Integrating the QMR Method with First Principles Material Science Application Code
First principles methods are used to aid the material designers and metallurgists in the investigation and design of new materials. These methods suffer from a divergent O(N{sup 3}) scaling which restricts the problem sizes that can be addressed. It will be demonstrated that the O(N{sup 3}) scaling problem can be overcome by employing real-space multiple scattering techniques to calculate the scattering properties, combined with the QMR algorithm. By incorporating the QMR method into our real-space multiple scattering code, we observed O(N) scaling for the large problem sizes of interest. This advancement will provide researchers with the necessary tools to treat large systems.