A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Boosting is a general method for improving the accuracy of any given learning algorithm. Focusing primarily on the AdaBoost algorithm, this chapter overviews some of the recent work on boosting including analyses of AdaBoost’s training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of AdaBoost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme. <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Decision trees are attractive classifiers due to their high execution speed. But trees derived with traditional methods often cannot be grown to arbitrary complexity for possible loss of generalization accuracy on unseen data. The limitation on complexity usually means suboptimal accuracy on training data. Following the principles of stochastic modeling, we propose a method to construct tree-based classifiers whose capacity can be arbitrarily expanded for increases in accuracy for both training and unseen data. The essence of the method is to build multiple trees in randomly selected subspaces of the feature space. Trees in, different subspaces generalize their classification in complementary ways, and their combined classification can be monotonically improved. The validity of the method is demonstrated through experiments on the recognition of handwritten digits. <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> It is shown that partial classification which allows for indecision in certain regions of the data space, can increase a benefit function, defined as the difference between the probabilities of correct and incorrect decisions, joint with the event that a decision is made. This is particularly true for small data samples, which may cause a large deviation of the estimated separation surface from the intersection surface between the corresponding probability density functions. Employing a particular density estimation method, an indecision domain is naturally defined by a single parameter whose optimal size, maximizing the benefit function, is derived from the data. The benefit function is shown to translate into profit in stock trading. Employing medical and economic data, it is shown that partial classification produces, on average, higher benefit values than full classification, assigning each new object to a class, and that the marginal benefit of partial classification reduces as the data size increases. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Combining multiple classifiers is an effective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding, that significantly improve classifiers like decision trees, rule learners, or neural networks. Unfortunately, these combining methods do not improve the nearest neighbor classifier. In this paper, we present MFS, a combining algorithm designed to improve the accuracy of the nearest neighbor NN classifier. MFS combines multiple NN classifiers each using only a random subset of features. The experimental results are encouraging: On 25 datasets from the UCI repository, MFS significantly outperformed several standard NN variants and was competitive with boosted decision trees. In additional experiments, we show that MFS is robust to irrelevant features, and is able to reduce both bias and variance components of error. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> A fruitful direction for future data mining research will be the development of techniques that incorporate privacy concerns. Specifically, we address the following question. Since the primary task in data mining is the development of models about aggregated data, can we develop accurate models without access to precise information in individual data records? We consider the concrete case of building a decision-tree classifier from training data in which the values of individual records have been perturbed. The resulting data records look very different from the original records and the distribution of data values is also very different from the original distribution. While it is not possible to accurately estimate original values in individual data records, we propose a novel reconstruction procedure to accurately estimate the distribution of original data values. By using these reconstructed distributions, we are able to build classifiers whose accuracy is comparable to the accuracy of classifiers built with the original data. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In this paper we propose a multi-expert architecture, particularly suited for verification systems, which attempts to offer the performance advantages of a serial approach while retaining the reliability of a parallel combination scheme. In this framework, criteria for evaluating the reliability of both the intermediate answers and the final response are provided, together with a method for the determination of an optimal reject option. This architecture has been tested on a signature verification application, confirming the effectiveness of the proposed approach. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> We devise a simple clustering-and-selection algorithm based on a probabilistic interpretation of classifier selection. First, the data set is clustered into K clusters, and then the most successful classifier for a given cluster is nominated to label the inputs in the Voronoi cell of the cluster centroid. The proposed method is compared experimentally with the minimum, maximum, product and average. Also given are the results from the naive Bayes method, the behaviour-knowledge space (BKS) method, the best individual and the oracle. <s> BIB010 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> This paper presents an efficient fuzzy classifier with the ability of feature selection based on a fuzzy entropy measure. Fuzzy entropy is employed to evaluate the information of pattern distribution in the pattern space. With this information, we can partition the pattern space into nonoverlapping decision regions for pattern classification. Since the decision regions do not overlap, both the complexity and computational load of the classifier are reduced and thus the training time and classification time are extremely short. Although the decision regions are partitioned into nonoverlapping subspaces, we can achieve good classification performance since the decision regions can be correctly determined via our proposed fuzzy entropy measure. In addition, we also investigate the use of fuzzy entropy to select relevant features. The feature selection procedure not only reduces the dimensionality of a problem but also discards noise-corrupted, redundant and unimportant features. Finally, we apply the proposed classifier to the Iris database and Wisconsin breast cancer database to evaluate the classification performance. Both of the results show that the proposed classifier can work well for the pattern classification application. <s> BIB011 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Ensemble methods have recently garnered a great deal of attention in the machine learning community. Techniques such as Boosting and Bagging have proven to be highly effective but require repeated resampling of the training data, making them inappropriate in a data mining context. The methods presented in this paper take advantage of plentiful data, building separate classifiers on sequential chunks of training points. These classifiers are combined into a fixed-size ensemble using a heuristic replacement strategy. The result is a fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift. <s> BIB012 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> This paper studies how to build a decision tree classifier under the following scenario: a database is vertically partitioned into two pieces, with one piece owned by Alice and the other piece owned by Bob. Alice and Bob want to build a decision tree classifier based on such a database, but due to the privacy constraints, neither of them wants to disclose their private pieces to the other party or to any third party.We present a protocol that allows Alice and Bob to conduct such a classifier building without having to compromise their privacy. Our protocol uses an untrusted third-party server, and is built upon a useful building block, the scalar product protocol. Our solution to the scalar product protocol is more efficient than any existing solutions. <s> BIB013 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Recently bagging, boosting and the random subspace method have become popular combining techniques for improving weak classifiers. These techniques are designed for, and usually applied to, decision trees. In this paper, in contrast to a common opinion, we demonstrate that they may also be useful in linear discriminant analysis. Simulation studies, carried out for several artificial and real data sets, show that the performance of the combining techniques is strongly affected by the small sample size properties of the base classifier: boosting is useful for large training sample sizes, while bagging and the random subspace method are useful for critical training sample sizes. Finally, a table describing the possible usefulness of the combining techniques for linear classifiers is presented. <s> BIB014 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Recently, mining data streams with concept drifts for actionable insights has become an important and challenging task for a wide range of applications including credit card fraud protection, target marketing, network intrusion detection, etc. Conventional knowledge discovery tools are facing two challenges, the overwhelming volume of the streaming data, and the concept drifts. In this paper, we propose a general framework for mining concept-drifting data streams using weighted ensemble classifiers. We train an ensemble of classification models, such as C4.5, RIPPER, naive Beyesian, etc., from sequential chunks of the data stream. The classifiers in the ensemble are judiciously weighted based on their expected classification accuracy on the test data under the time-evolving environment. Thus, the ensemble approach improves both the efficiency in learning the model and the accuracy in performing classification. Our empirical study shows that the proposed methods have substantial advantage over single-classifier approaches in prediction accuracy, and the ensemble framework is effective for a variety of classification models. <s> BIB015 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Algorithms for tracking concept drift are important for many applications. We present a general method based on the weighted majority algorithm for using any online learner for concept drift. Dynamic weighted majority (DWM) maintains an ensemble of base learners, predicts using a weighted-majority vote of these "experts", and dynamically creates and deletes experts in response to changes in performance. We empirically evaluated two experimental systems based on the method using incremental naive Bayes and incremental tree inducer [ITI] as experts. For the sake of comparison, we also included Blum's implementation of weighted majority. On the STAGGER concepts and on the SEA concepts, results suggest that the ensemble method learns drifting concepts almost as well as the base algorithms learn each concept individually. Indeed, we report the best overall results for these problems to date. <s> BIB016 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. The generalization error for forests converges a.s. to a limit as the number of trees in the forest becomes large. The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them. Using a random selection of features to split each node yields error rates that compare favorably to Adaboost (Y. Freund & R. Schapire, Machine Learning: Proceedings of the Thirteenth International conference, aaa, 148–156), but are more robust with respect to noise. Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the splitting. Internal estimates are also used to measure variable importance. These ideas are also applicable to regression. <s> BIB017 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In this work, the authors have evaluated almost 20 millions ensembles of classifiers generated by several methods. Trying to optimize those ensembles based on the nearest neighbours and the random subspaces paradigms, we found that the use of a diversity metric called "ambiguity" had no better positive impact than plain stochastic search. <s> BIB018 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Abstract Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally. <s> BIB019 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy. <s> BIB020 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Dynamic classifier selection (DCS) plays a strategic role in the field of multiple classifier systems (MCS). This paper proposes a study on the performances of DCS by Local Accuracy estimation (DCS-LA). To this end, upper bounds against which the performances can be evaluated are proposed. The experimental results on five datasets clearly show the effectiveness of the selection methods based on local accuracy estimates. <s> BIB021 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. One popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. In this paper, we propose a new dynamic classifier selection (DCS) mechanism to integrate base classifiers for effective mining from data streams. The proposed algorithm dynamically selects a single “best” classifier to classify each test instance at run time. Our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. Given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. Experimental results and comparative studies demonstrate the efficiency and efficacy of our method. Such a DCS scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence. <s> BIB022 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Online signature verification systems based on one-class classifiers are presented. Global information is extracted with a feature-based representation and recognized by using an ensemble of one-class classifiers. Experimental results obtained on the SUBCORPUS-100 MCYT signature database (100 signers, 5000 signatures) show that the machine experts, here proposed, outperform the state-of-the-art works both for random and skilled forgeries. <s> BIB023 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Relevance feedback schemes based on support vector machines (SVM) have been widely used in content-based image retrieval (CBIR). However, the performance of SVM-based relevance feedback is often poor when the number of labeled positive feedback samples is small. This is mainly due to three reasons: 1) an SVM classifier is unstable on a small-sized training set, 2) SVM's optimal hyperplane may be biased when the positive feedback samples are much less than the negative feedback samples, and 3) overfitting happens because the number of feature dimensions is much higher than the size of the training set. In this paper, we develop a mechanism to overcome these problems. To address the first two problems, we propose an asymmetric bagging-based SVM (AB-SVM). For the third problem, we combine the random subspace method and SVM for relevance feedback, which is named random subspace SVM (RS-SVM). Finally, by integrating AB-SVM and RS-SVM, an asymmetric bagging and random subspace SVM (ABRS-SVM) is built to solve these three problems and further improve the relevance feedback performance <s> BIB024 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> We present attribute bagging (AB), a technique for improving the accuracy and stability of classifier ensembles induced using random subsets of features. AB is a wrapper method that can be used with any learning algorithm. It establishes an appropriate attribute subset size and then randomly selects subsets of features, creating projections of the training set on which the ensemble classifiers are built. The induced classifiers are then used for voting. This article compares the performance of our AB method with bagging and other algorithms on a hand-pose recognition dataset. It is shown that AB gives consistently better results than bagging, both in accuracy and stability. The performance of ensemble voting in bagging and the AB method as a function of the attribute subset size and the number of voters for both weighted and unweighted voting is tested and discussed. We also demonstrate that ranking the attribute subsets by their classification accuracy and voting using only the best subsets further improves the resulting performance of the ensemble. <s> BIB025 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> As data streams are gaining prominence in a growing number of emerging application domains, classification on data streams is becoming an active research area. Currently, the typical approach to this problem is based on ensemble learning, which learns basic classifiers from training data stream and forms the global predictor by organizing these basic ones. While this approach seems successful to some extent, its performance usually suffers from two contradictory elements existing naturally within many application scenarios: firstly, the need for gathering sufficient training data for basic classifiers and engaging enough basic learners in voting for bias-variance reduction; and secondly, the requirement for significant sensitivity to concept-drifts, which places emphasis on using recent training data and up-to-date individual classifiers. It results in such a dilemma that some algorithms are not sensitive enough to concept-drifts while others, although sensitive enough, suffer from unsatisfactory classification accuracy. In this paper, we propose an ensemble learning algorithm, which: (1) furnishes training data for basic classifiers, starting from the up-to-date data chunk and searching for complement from past chunks while ruling out the data inconsistent with current concept; (2) provides effective voting by adaptively distinguishing sensible classifiers from the else and engaging sensible ones as voters. Experimental results justify the superiority of this strategy in terms of both accuracy and sensitivity, especially in severe circumstances where training data is extremely insufficient or concepts are evolving frequently and significantly. <s> BIB026 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> The paper presents the novel adaptive splitting and selection algorithm (AdaSS) used for learning compound pattern recognition system. Splitting a feature space into its constituents and selection of the best area classifier from the pool of available recognizers for each region are key processes of the proposed model. Both take place simultaneously as part of a compound optimization process aimed at maximizing system performance. Evolutionary algorithms are used to find out the optimal solution. The results of experiments for algorithm evaluation purposes prove the quality of the proposed approach. <s> BIB027 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> Fingerprint classification reduces the number of possible matches in automated fingerprint identification systems by categorizing fingerprints into predefined classes. Support vector machines (SVMs) are widely used in pattern classification and have produced high accuracy when performing fingerprint classification. In order to effectively apply SVMs to multi-class fingerprint classification systems, we propose a novel method in which the SVMs are generated with the one-vs-all (OVA) scheme and dynamically ordered with nai@?ve Bayes classifiers. This is necessary to break the ties that frequently occur when working with multi-class classification systems that use OVA SVMs. More specifically, it uses representative fingerprint features as the FingerCode, singularities and pseudo ridges to train the OVA SVMs and nai@?ve Bayes classifiers. The proposed method has been validated on the NIST-4 database and produced a classification accuracy of 90.8% for five-class classification with the statistical significance. The results show the benefits of integrating different fingerprint features as well as the usefulness of the proposed method in multi-class fingerprint classification. <s> BIB028 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In handwritten pattern recognition, the multiple classifier system has been shown to be useful for improving recognition rates. One of the most important tasks in optimizing a multiple classifier system is to select a group of adequate classifiers, known as an Ensemble of Classifiers (EoC), from a pool of classifiers. Static selection schemes select an EoC for all test patterns, and dynamic selection schemes select different classifiers for different test patterns. Nevertheless, it has been shown that traditional dynamic selection performs no better than static selection. We propose four new dynamic selection schemes which explore the properties of the oracle concept. Our results suggest that the proposed schemes, using the majority voting rule for combining classifiers, perform better than the static selection method. <s> BIB029 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In the context of Ensembles or Multi-Classifier Systems, the choice of the ensemble members is a very complex task, in which, in some cases, it can lead to ensembles with no performance improvement. In order to avoid this situation, there is a great deal of research to find effective classifier member selection methods. In this paper, we propose a selection criterion based on both the accuracy and diversity of the classifiers in the initial pool. Also, instead of using a static selection method, we use a Dynamic Classifier Selection (DSC) procedure. In this case, the member classifiers to form the ensemble are chosen at the test (use) phase. That is, different testing patterns can be classified by different ensemble configurations. <s> BIB030 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In the real world concepts are often not stable but change with time. A typical example of this in the biomedical context is antibiotic resistance, where pathogen sensitivity may change over time as new pathogen strains develop resistance to antibiotics that were previously effective. This problem, known as concept drift, complicates the task of learning a model from data and requires special approaches, different from commonly used techniques that treat arriving instances as equally important contributors to the final concept. The underlying data distribution may change as well, making previously built models useless. This is known as virtual concept drift. Both types of concept drifts make regular updates of the model necessary. Among the most popular and effective approaches to handle concept drift is ensemble learning, where a set of models built over different time periods is maintained and the best model is selected or the predictions of models are combined, usually according to their expertise level regarding the current concept. In this paper we propose the use of an ensemble integration technique that would help to better handle concept drift at an instance level. In dynamic integration of classifiers, each base classifier is given a weight proportional to its local accuracy with regard to the instance tested, and the best base classifier is selected, or the classifiers are integrated using weighted voting. Our experiments with synthetic data sets simulating abrupt and gradual concept drifts and with a real-world antibiotic resistance data set demonstrate that dynamic integration of classifiers built over small time intervals or fixed-sized data blocks can be significantly better than majority voting and weighted voting, which are currently the most commonly used integration techniques for handling concept drift with ensembles. <s> BIB031 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> There is currently considerable enthusiasm around the MapReduce (MR) paradigm for large-scale data analysis [17]. Although the basic control flow of this framework has existed in parallel SQL database management systems (DBMS) for over 20 years, some have called MR a dramatically new computing model [8, 17]. In this paper, we describe and compare both paradigms. Furthermore, we evaluate both kinds of systems in terms of performance and development complexity. To this end, we define a benchmark consisting of a collection of tasks that we have run on an open source version of MR as well as on two parallel DBMSs. For each task, we measure each system's performance for various degrees of parallelism on a cluster of 100 nodes. Our results reveal some interesting trade-offs. Although the process to load data into and tune the execution of parallel DBMSs took much longer than the MR system, the observed performance of these DBMSs was strikingly better. We speculate about the causes of the dramatic performance difference and consider implementation concepts that future systems should take from both kinds of architectures. <s> BIB032 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> This paper introduces a new ensemble approach, Feature-Subspace Aggregating (Feating), which builds local models instead of global models. Feating is a generic ensemble approach that can enhance the predictive performance of both stable and unstable learners. In contrast, most existing ensemble approaches can improve the predictive performance of unstable learners only. Our analysis shows that the new approach reduces the execution time to generate a model in an ensemble through an increased level of localisation in Feating. Our empirical evaluation shows that Feating performs significantly better than Boosting, Random Subspace and Bagging in terms of predictive accuracy, when a stable learner SVM is used as the base learner. The speed up achieved by Feating makes feasible SVM ensembles that would otherwise be infeasible for large data sets. When SVM is the preferred base learner, we show that Feating SVM performs better than Boosting decision trees and Random Forests. We further demonstrate that Feating also substantially reduces the error of another stable learner, k-nearest neighbour, and an unstable learner, decision tree. <s> BIB033 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> The paper focuses on the problem of privacy preserving for classification task. This issue is quite an important subject for the machine learning approach based on distributed databases. On the basis of the study of available works devoted to privacy we propose its new definition and its taxonomy. We use this taxonomy to create several modifications of k-nearest neighbors classifier which are consistent with the proposed privacy levels. Their computational complexity are evaluated on the basis of computer experiments. <s> BIB034 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> The growing amount of various kinds of information triggers the need to develop efficient network computing systems, as single machines in many cases are not able to provide effective processing and analysis. One of the very promising approaches of distributed data analysis is combined classification, which could be relatively easily implemented in distributed computing systems. In this paper we address problem of decentralized distributed computing system for mentioned above classification method. We focus on the system fairness. The performance metric is defined as a maximum response time, i.e., the computing system should be designed to minimize the response time of each client using the system. We assume that the system is decentralized and each request is sent by the client directly to computing nodes without assistance of a central service. An ILP (Integer Linear Programming) model is formulated and applied to obtain optimal results provided by branch-and-cut algorithm included in the CPLEX solver. Widespread simulations are performed to evaluate properties of the computing system in terms of several parameters describing the system. <s> BIB035 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> This study presents a novel hybrid intelligent system using both unsupervised and supervised learning that can be easily adapted to be used in an individual or collaborative system. The system divides the classification problem into two stages: firstly it divides the input data space into different parts, according to the input space distribution of the data set. Then, it generates several simple classifiers that are used to correctly classify samples that are contained in one of the previously determined parts. This way, the efficiency of each classifier increases, as they can specialize in classifying only related samples from certain regions of the input data space. This specialization of the single classifiers enables them to learn more specific patterns or characteristics of the data space, avoiding the risk of obtaining a general algorithm that over-fits to the data. The hybrid system presented has been tested with artificial and real data sets. A comparative study of the results obtained by the novel model with those obtained from other common classification methods is also included in the present work. <s> BIB036 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> The concept of a classifier competence is fundamental to multiple classifier systems (MCSs). In this study, a method for calculating the classifier competence is developed using a probabilistic model. In the method, first a randomised reference classifier (RRC) whose class supports are realisations of the random variables with beta probability distributions is constructed. The parameters of the distributions are chosen in such a way that, for each feature vector in a validation set, the expected values of the class supports produced by the RRC and the class supports produced by a modelled classifier are equal. This allows for using the probability of correct classification of the RRC as the competence of the modelled classifier. The competences calculated for a validation set are then generalised to an entire feature space by constructing a competence function based on a potential function model or regression. Three systems based on a dynamic classifier selection and a dynamic ensemble selection (DES) were constructed using the method developed. The DES based system had statistically significant higher average rank than the ones of eight benchmark MCSs for 22 data sets and a heterogeneous ensemble. The results obtained indicate that the full vector of class supports should be used for evaluating the classifier competence as this potentially improves performance of MCSs. <s> BIB037 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> This paper presents a significant modification to the AdaSS Adaptive Splitting and Selection algorithm, which was developed several years ago. The method is based on the simultaneous partitioning of the feature space and an assignment of a compound classifier to each of the subsets. The original version of the algorithm uses a classifier committee and a majority voting rule to arrive at a decision. The proposed modification replaces the fairly simple fusion method with a combined classifier, which makes a decision based on a weighted combination of the discriminant functions of the individual classifiers selected for the committee. The weights mentioned above are dependent not only on the classifier identifier, but also on the class number. The proposed approach is based on the results of previous works, where it was proven that such a combined classifier method could achieve significantly better results than simple voting systems. The proposed modification was evaluated through computer experiments, carried out on diverse benchmark datasets. The results are very promising in that they show that, for most of the datasets, the proposed method outperforms similar techniques based on the clustering and selection approach. <s> BIB038 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying input data <s> In this paper, a measure of competence based on random classification (MCR) for classifier ensembles is presented. The measure selects dynamically (i.e. for each test example) a subset of classifiers from the ensemble that perform better than a random classifier. Therefore, weak (incompetent) classifiers that would adversely affect the performance of a classification system are eliminated. When all classifiers in the ensemble are evaluated as incompetent, the classification accuracy of the system can be increased by using the random classifier instead. Theoretical justification for using the measure with the majority voting rule is given. Two MCR based systems were developed and their performance was compared against six multiple classifier systems using data sets taken from the UCI Machine Learning Repository and Ludmila Kuncheva Collection. The systems developed had typically the highest classification accuracies regardless of the ensemble type used (homogeneous or heterogeneous). <s> BIB039
This diversification strategy assumes that classifiers trained on different (disjoint) input subspaces become complementary. Three general strategies are identified: 1. Using different data partitions. 2. Using different sets of features. 3. Taking into consideration the local specialization of individual classifiers. Data partitions They may be compelled by several reasons, such as data privacy, or the need to learn over distributed data chunks stored in different databases BIB013 BIB034  . Regarding data privacy, we should notice that using distributed data may come up against legal or commercial constraints which do not allow sharing raw datasets and merging them into a common repository BIB007 . To ensure privacy we can train individual classifiers on each database independently and merge their outputs using hybrid classifier principles BIB035 . The distributed data paradigm is strongly connected with the big data analysis problem BIB032 . A huge database may impede to deliver trained classifiers under specified time constraints, imposing to resort to sampling techniques to obtain manageable dataset partitions. A well known approach is crossvalidated committee which requires to minimize overlapping of dataset partitions BIB002 . Providing individualized train datasets for each classifier is convenient in the case of shortage of learning examples. Most popular techniques, such as bagging BIB020 or boosting    BIB001 , have their origin in bootstrapping  . These methods try to ascertain if a set of weak classifier may produce a strong one. Bagging applies sampling with replacement to obtain independent training datasets for each individual classifier. Boosting modifies the input data distribution perceived by each classifier from the results of classifiers trained before, focusing on difficult samples, making the final decision by a weighted voting rule. Data features May be selected to ensure diversity training of a pool of classifiers. The Random Subspace BIB003 BIB004 was employed for several types of the individual classifiers such as decision tree (Random Forest) BIB017 , linear classifiers BIB014 , or minimal distance classifier BIB018 BIB006 . It is worth pointing out the interesting propositions dedicated one-class classifier presented by Nanni BIB023 or an hierarchical method of ensemble forming, based on feature space splitting and then assigning two-class classifiers (i.e. Support Vector Machines) locally presented in BIB024 BIB033 . Attribute Bagging BIB025 is a wrapper method that establishes the appropriate size of a feature subset, and then creates random projections of a given training set by random selection of feature subsets. The classifier ensemble are train on the basis of the obtained set. Local specialization It is assumed for classifier selection, selecting the best single classifier from a pool of classifiers trained over each partition of the feature space. It gives the MCS answer for all objects included in the partition  . Some proposals assume classifier local specialization, providing only locally optimal solutions BIB008 BIB005 BIB009  BIB019 , while others divide the feature space, selecting (or training) a classifier for each partition. Static and dynamic approaches are distinguished: Static classifier selection BIB036 : the relation between region of competence and assigned classifier is fixed. Kuncheva's Clustering and Selection algorithm BIB010 partitions the feature space by a clustering algorithm, and selects the best individual classifier for each cluster according to its local accuracy. Adaptive Splitting and Selection algorithm in BIB027 partitions the feature space and assigns classifiers to each partition into one integrated process. The main advantage of AdaSS is that the training algorithm considers an area contour to determine the classifier content and, conversely, that the region shapes adapt to the competencies of the classifiers. Additionally, the majority voting or more sophisticated rules are proposed as combination method of area classifiers BIB038 . Lee et al. BIB011 used the fuzzy entropy measure to partition the feature space and select the relevant features with good separability for each of them. Dynamic classifier selection: the competencies of the individual classifiers are calculated during classification operation BIB028 BIB029 BIB021  . There are several interesting proposals which extend this concept, e.g., by using preselected committee of the individual classifier and making the final decision on the basis of a voting rule BIB030 . In BIB037 BIB039 authors propose dynamic ensemble selection based on the original competence measure using classification of so-called random reference classifier. Both static BIB012 BIB015 BIB026 and dynamic BIB016 BIB031 BIB022 classifier specialization are widely used for data stream classification.