A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Consider a group of people confronted with a dichotomous choice (for example, a yes or no decision). Assume that we can characterize each person by a probability, pi, of making the 'better' of the two choices open to the group, such that we define 'better' in terms of some linear ordering of the alternatives. If individual choices are independent, and if the a priori likelihood that either of the two choices is correct is one half, we show that the group decision procedure that maximizes the likelihood that the group will make the better of the two choices open to it is a weighted voting rule that assigns weights, wi, such that <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. > <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Possible solutions to the problem of combining classifiers can be divided into three categories according to the levels of information available from the various classifiers. Four approaches based on different methodologies are proposed for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-nearest-neighbor, and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly. For example, on the US zipcode database, 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as high reliability with 95% recognition, 0% substitution, and 5% rejection. > <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> In pattern recognition, there is a growing use of multiple classifier combinations with the goal to increase recognition performance. In many cases, plurality voting is a part of the combination process. In this article, we discuss and test several well known voting methods from politics and economics on classifier combination in order to see if an alternative to the simple plurality vote exists. We found that, assuming a number of prerequisites, better methods are available, that are comparatively simple and fast. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Recently, multiple classifier systems (MCS) have been used for practical applications to improve classification accuracy. Self-generating neural networks (SGNN) are one of the suitable base-classifiers for MCS because of their simple setting and fast learning. However, the computation cost of the MCS increases in proportion to the number of SGNN. In this paper, we propose a novel optimization method for the structure of the SGNN in the MCS. We compare the optimized MCS with two sampling methods. Experiments have been conducted to compare the optimized MCS with an unoptimized MCS, the MCS based on C4.5, and k-nearest neighbor. The results show that the optimized MCS can improve its classification accuracy as well as reducing the computation cost. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> We derive upper and lower limits on the majority vote accuracy with respect to individual accuracy p, the number of classiers in the pool (L), and the pairwise dependence between classiers, measured by Yule's Q statistic. Independence between individual classi- ers is typically viewed as an asset in classier fusion. We show that the majority vote with dependent classiers can potentially oer a dramatic improvement both over independent clas- siers and over the individual accuracy p. A functional relationship between the limits and the pairwise dependence Q is derived. Two patterns of the joint distribution for classier outputs (correct/incorrect) are identied to derive the limits: the pattern of success and the pattern of failure. The results support the intuition that negative pairwise dependence is benecial although not straightforwardly related to the accuracy. The pattern of success showed that for the highest improvement over p, all pairs of classiers in the pool should have the same negative dependence. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> A wide selection of standard statistical pattern classification algorithms can be applied as trainable fusion rules while designing neural network ensembles. A focus of the present two-part paper is finite sample effects: the complexity of base classifiers and fusion rules; the type of outputs provided by experts to the fusion rule; non-linearity of the fusion rule; degradation of experts and the fusion rule due to the lack of information in the design set; the adaptation of base classifiers to training set size, etc. In the first part of this paper, we consider arguments for utilizing continuous outputs of base classifiers versus categorical outputs and conclude: if one succeeds in having a small number of expert networks working perfectly in different parts of the input feature space, then crisp outputs may be preferable over continuous outputs. Afterwards, we oppose fixed fusion rules versus trainable ones and demonstrate situations where weighted average fusion can outperform simple average fusion. We present a review of statistical classification rules, paying special attention to these linear and non-linear rules, which are employed rarely but, according to our opinion, could be useful in neural network ensembles. We consider ideal and sample-based oracle decision rules and illustrate characteristic features of diverse fusion rules by considering an artificial two-dimensional (2D) example where the base classifiers perform well in different regions of input feature space. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Profound theoretical analysis is performed of small-sample properties of trainable fusion rules to determine in which situations neural network ensembles can improve or degrade classification results. We consider small sample effects, specific only to multiple classifiers system design in the two-category case of two important fusion rules: (1) linear weighted average (weighted voting), realized either by the standard Fisher classifier or by the single-layer perceptron, and (2) the non-linear Behavior-Knowledge-Space method. The small sample effects include: (i) training bias, i.e. learning sample size influence on generalization error of the base experts or of the fusion rule, (ii) optimistic biased outputs of the experts (self-boasting effect) and (iii) sample size impact on determining optimal complexity of the fusion rule. Correction terms developed to reduce the self-boasting effect are studied. It is shown that small learning sets increase classification error of the expert classifiers and damage correlation structure between their outputs. If the sizes of learning sets used to develop the expert classifiers are too small, non-trainable fusion rules can outperform more sophisticated trainable ones. A practical technique to fight sample size problems is a noise injection technique. The noise injection reduces the fusion rule's complexity and diminishes the expert's boasting bias. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> This book, which is wholly devoted to the subject of model combination, is divided into ten chapters. In addition to the first two introductory chapters, the book covers some of the following topics: multiple classifier systems; combination methods when the base classifier outputs are 0/1; methods when the outputs are continuous, e.g., posterior probabilities; methods for classifier selection; bagging and boosting; the theory of fixed combination rules; and the concept of diversity. Overall, it is a very well-written monograph. It explains and analyzes different approaches comparatively so that the reader can see how they are similar and how they differ. The literature survey is extensive. The MATLAB code for many methods is given in chapter appendices allowing readers to play with the explained methods or apply them quickly to their own data. The book is a must-read for researchers and practitioners alike. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Class label fusion <s> Multiple Classifier Systems are nowadays one of the most promising directions in pattern recognition. There are many methods of decision making by the ensemble of classifiers. The most popular are methods that have their origin in voting method, where the decision of the common classifier is a combination of individual classifiers' outputs. This work presents comparative analysis of some classifier fusion methods based on weighted voting of classifiers' responses and combination of classifiers' discriminant functions. We discus which of presented methods could produce classifier better than Oracle one. Some results of computer experiments carried out on benchmark and computer generated data which confirmed our studies are presented also. <s> BIB010
Early algorithms performing fusion of classifier responses BIB003  BIB006 only implemented majority voting schemes in three main versions BIB009 : unanimous voting, so that the answer requires that all classifiers agree, simple majority, so that the answer is given if majority is greater than half the pool of classifiers, majority voting, taking the answer with the highest number of votes. The expected error of majority voting (for independent classifiers with the same quality) was estimated in 1794 according to Bernoulli's equation, proven as the Condorcet Jury Theorem BIB001 . Later works focused on the analytically derived classification performance of combined classifiers hold only when strong conditions are met BIB002 so that they are not useful from practical point of view. Alternative voting methods weight differently the decisions coming from different committee members BIB009 BIB004 . The typical architecture of combined classifier based on class labels is presented in the left diagram of Fig. 6 . In BIB010 authors distinguished the types of weighted voting according to the classifier, both to the classifier and the class, and, finally, to features values, the classifier and the class. Anyway, no one of these models can improve over the Oracle. To achieve that we need additional information, such as the feature values BIB007 BIB008 BIB005 as depicted in the right diagram of Fig. 6 .