A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> We present a new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases. The new procedure can be viewed either as a modular version of a multilayer supervised network, or as an associative version of competitive learning. It therefore provides a new link between these two apparently different approaches. We demonstrate that the learning procedure divides up a vowel discrimination task into appropriate subtasks, each of which can be solved by a very simple expert network. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd. <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> A multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures. Decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem. The rankings can be combined by methods that either reduce or rerank a given set of classes. An intersection method and union method are proposed for class set reduction. Three methods based on the highest rank, the Borda count, and logistic regression are proposed for class set reranking. These methods have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness. > <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> This paper discusses the linearly weighted combination of estimators in which the weighting functions are dependent on the input. We show that the weighting functions can be derived either by evaluating the input dependent variance of each estimator or by estimating how likely it is that a given estimator has seen data in the region of the input space close to the input pattern. The latter solution is closely related to the mixture of experts approach and we show how learning rules for the mixture of experts can be derived from the theory about learning with missing features. The presented approaches are modular since the weighting functions can easily be modified (no retraining) if more estimators are added. Furthermore, it is easy to incorporate estimators which were not derived from data such as expert systems or algorithms. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> In a classification problem, improved accuracy can be obtained in many situations by using the combination of several classifiers instead of a single one. Turner and Gosh (1999) derived the error reduction that can be obtained by combining unbiased classifiers with independent errors using a simple average. We present an extension of this result by finding the improvement obtained when combining classifiers using weighted average. We also prove that for unbiased classifiers with independent errors the best combination of N classifiers corresponds to a weighted average, where the combination coefficient of each classifier is equal to 1/N. This means that in these cases the simple average should be used. We present experiments illustrating our results. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> When more than a single classifier has been trained for the same recognition problem the question arises how this set of classifiers may be combined into a final decision rule. Several fixed combining rules are used that depend on the output values of the base classifiers only. They are almost always suboptimal. Usually, however, training sets are available. They may be used to calibrate the base classifier outputs, as well as to build a trained combining classifier using these outputs as inputs. It depends on various circumstances whether this is useful, in particular whether the training set is used for the base classifiers as well and whether they are overtrained. We present an intuitive discussion on the use of trained combiners, relating the question of the choice of the combining classifier to a similar choice in the area of dissimilarity based pattern recognition. Some simple examples are used to illustrate the discussion. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> Amidst the conflicting experimental evidence of superiority of one over the other, we investigate the Sum and majority Vote combining rules in a two class case, under the assumption of experts being of equal strength and estimation errors conditionally independent and identically distributed. We show, analytically, that, for Gaussian estimation error distributions, Sum always outperforms Vote. For heavy tail distributions, we demonstrate by simulation that Vote may outperform Sum. Results on synthetic data confirm the theoretical predictions. Experiments on real data support the general findings, but also show the effect of the usual assumptions of conditional independence, identical error distributions, and common target outputs of the experts not being fully satisfied. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> A generic fusion problem is studied for multiple sensors whose outputs are probabilistically related to their inputs according to unknown distributions. Sensor measurements are provided as iid input-output samples, and an empirical risk minimization method is described for designing fusers with distribution-free performance bounds. The special cases of isolation and projective fusers for classifiers and function estimators, respectively, are described in terms of performance bounds. The isolation fusers for classifiers are probabilistically guaranteed to perform at least as good as the best classifier. The projective fusers for function estimators are probabilistically guaranteed to perform at least as good as the best subset of estimators. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> This paper presents practical aspects of feature set decomposition in classification problems using decision trees. Feature set decomposition generalizes the task of feature selection which is extensively used in data mining. Feature selection aims to provide a representative set of features from which a classifier is constructed. On the other hand, feature set decomposition decomposes the original set of features into several subsets, and builds a classifier for each subset. The classifiers are then combined for classifying new instances. In order to examine the idea, a general framework that searches for helpful decomposition structures is proposed. This framework nests many algorithms, two of which are tested empirically over a set of benchmark datasets. The first algorithm performs a serial search while using a new Vapnik-Chervonenkis dimension bound for multiple oblivious trees as an evaluating schema. The second algorithm performs a multi-search while using wrapper evaluating schema. This work indicates that feature set decomposition can increase the accuracy of decision trees. <s> BIB010 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> In this paper, a theoretical and experimental analysis of linear combiners for multiple classifier systems is presented. Although linear combiners are the most frequently used combining rules, many important issues related to their operation for pattern classification tasks lack a theoretical basis. After a critical review of the framework developed in works by Turner and Ghosh [1996], [1999] on which our analysis is based, we focus on the simplest and most widely used implementation of linear combiners, which consists of assigning a nonnegative weight to each individual classifier. Moreover, we consider the ideal performance of this combining rule, i.e., that achievable when the optimal values of the weights are used. We do not consider the problem of weights estimation, which has been addressed in the literature. Our theoretical analysis shows how the performance of linear combiners, in terms of misclassification probability, depends on the performance of individual classifiers, and on the correlation between their outputs. In particular, we evaluate the ideal performance improvement that can be achieved using the weighted average over the simple average combining rule and investigate in what way it depends on the individual classifiers. Experimental results on real data sets show that the behavior of linear combiners agrees with the predictions of our analytical model. Finally, we discuss the contribution to the state of the art and the practical relevance of our theoretical and experimental analysis of linear combiners for multiple classifier systems. <s> BIB011 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> A new theoretical framework for the analysis of linear combiners is presented in this paper. This framework extends the scope of previous analytical models, and provides some new theoretical results which improve the understanding of linear combiners operation. In particular, we show that the analytical model developed in seminal works by Tumer and Ghosh is included in this framework. <s> BIB012 </s> A survey of multiple classifier systems as hybrid systems <s> Support function fusion <s> Multiple Classifier Systems are nowadays one of the most promising directions in pattern recognition. There are many methods of decision making by the ensemble of classifiers. The most popular are methods that have their origin in voting method, where the decision of the common classifier is a combination of individual classifiers' outputs. This work presents comparative analysis of some classifier fusion methods based on weighted voting of classifiers' responses and combination of classifiers' discriminant functions. We discus which of presented methods could produce classifier better than Oracle one. Some results of computer experiments carried out on benchmark and computer generated data which confirmed our studies are presented also. <s> BIB013
Support function fusion system architecture is depicted in Fig. 7 . Support functions provide a score for the decision taken by an individual classifier. The value of a support function is the estimated likelihood of a class, computed either as a neural network output, a posteriori probability, or fuzzy membership function. First to be mentioned, the Borda count BIB003 computes an score for each class on the basis of its ranking by each individual classifier. The most popular form of support function is the a posteriori probability  , produced by the probabilistic models embodied by the classifiers BIB006 BIB012 BIB008 . There are many works following this approach, such as the optimal projective fuser of BIB009 , the combination of neural networks outputs according to their accuracy BIB005 , and Naïve Bayes as the MCS combination method BIB010 . Some analytical properties and experimental evaluations of aggregating methods were presented in  BIB002 BIB011  . The aggregating methods use simple operators such as supremum or the mean value. They do not involve learning. However, they have little practical applicability because of the hard conditions imposed by them BIB007 . The main aggregating advantage is that it counteracts over-fitting of individual classifiers. According to BIB013 , the following types of weighted aggregation can be identified depending on: (a) only the classifier id, (b) the classifier and the feature vector, (c) on the classifier and the class, and (d) on the classifier, the class, and the feature vector. For two-class recognition problems only the last two types of aggregation allow to produce compound classifier which may improve the Oracle. For many-class problems, it is possible to improve the Oracle  using any of these aggregation methods. Finally, another salient approach is the mixture of experts BIB001  which combines classifier outputs using so-called input dependent gating function. Tresp and Taniguchi BIB004 proposed a linear function for this fuser model, and Cheeseman  proposed a mixture of Gaussian.