A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Consider a reliability study in which different subjects are judged on a dichotomous trait by dif ferent sets of judges, possibly unequal in number. A kappa-like measure of reliability is proposed, its correspondence to an intraclass correlation co efficient is pointed out, and a test for its statistical significance is presented. A numerical example is given. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. > <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme. <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Abstract The topic of this paper is the exploitation of diversity to enhance computer system reliability. It is well established that a diverse system composed of multiple alternative versions is more reliable than any single version alone, and this knowledge has occasionally been exploited in safety-critical applications. However, it is not clear what this property is, nor how the available diversity in a collection of versions is best exploited. We develop, define, illustrate and assess diversity measures, voting strategies for diversity exploitation, and interactions between the two. We take the view that a proper understanding of such issues is required if multiversion software engineering is to be elevated from the current “try it and see” procedure to a systematic technology. In addition, we introduce inductive programming techniques, particularly neural computing, as a cost-effective route to the practical use of multiversion systems outside the demanding requirements of safety-critical systems, i.e. in general software engineering. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> An appropriate use of neural computing techniques is to apply them to problems such as condition monitoring, fault diagnosis, control and sensing, where conventional solutions can be hard to obtain. However, when neural computing techniques are used, it is important that they are employed so as to maximise their performance, and improve their reliability. Their performance is typically assessed in terms of their ability to generalise to a previously unseen test set, although unless the training set is very carefully chosen, 100p accuracy is rarely achieved. Improved performance can result when sets of neural nets are combined in ensembles and ensembles can be viewed as an example of the reliability through redundancy approach that is recommended for conventional software and hardware in safety-critical or safety-related applications. Although there has been recent interest in the use of neural net ensembles, such techniques have yet to be applied to the tasks of condition monitoring and fault diagnosis. In this paper, we focus on the benefits of techniques which promote diversity amongst the members of an ensemble, such that there is a minimum number of coincident failures. The concept of ensemble diversity is considered in some detail, and a hierarchy of four levels of diversity is presented. This hierarchy is then used in the description of the application of ensemble-based techniques to the case study of fault diagnosis of a diesel engine. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Feature subset-selection has emerged as a useful technique for creating diversity in ensembles - particularly in classification ensembles. In this paper we argue that this diversity needs to be monitored in the creation of the ensemble. We propose an entropy measure of the outputs of the ensemble members as a useful measure of the ensemble diversity. Further, we show that using the associated conditional entropy as a loss function (error measure) works well and the entropy in the ensemble predicts well the reduction in error due to the ensemble. These measures are evaluated on a medical prediction problem and are shown to predict the performance of the ensemble well. We also show that the entropy measure of diversity has the added advantage that it seems to model the change in diversity with the size of the ensemble. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Abstract In the field of pattern recognition, the combination of an ensemble of neural networks has been proposed as an approach to the development of high performance image classification systems. However, previous work clearly showed that such image classification systems are effective only if the neural networks forming them make different errors. Therefore, the fundamental need for methods aimed to design ensembles of ‘error-independent’ networks is currently acknowledged. In this paper, an approach to the automatic design of effective neural network ensembles is proposed. Given an initial large set of neural networks, our approach is aimed to select the subset formed by the most error-independent nets. Reported results on the classification of multisensor remote-sensing images show that this approach allows one to design effective neural network ensembles. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Abstract This study looks at the relationships between different methods of classifier combination and different measures of diversity. We considered 10 combination methods and 10 measures of diversity on two benchmark data sets. The relationship was sought on ensembles of three classifiers built on all possible partitions of the respective feature sets into subsets of pre-specified sizes. The only positive finding was that the Double-Fault measure of diversity and the measure of difficulty both showed reasonable correlation with Majority Vote and Naive Bayes combinations. Since both these measures have an indirect connection to the ensemble accuracy, this result was not unexpected. However, our experiments did not detect a consistent relationship between the other measures of diversity and the 10 combination methods. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Neural network ensemble is a learning paradigm where many neural networks are jointly used to solve a problem. In this paper, the relationship between the ensemble and its component neural networks is analyzed from the context of both regression and classification, which reveals that it may be better to ensemble many instead of all of the neural networks at hand. This result is interesting because at present, most approaches ensemble all the available neural networks for prediction. Then, in order to show that the appropriate neural networks for composing an ensemble can be effectively selected from a set of available neural networks, an approach named GASEN is presented. GASEN trains a number of neural networks at first. Then it assigns random weights to those networks and employs genetic algorithm to evolve the weights so that they can characterize to some extent the fitness of the neural networks in constituting an ensemble. Finally it selects some neural networks based on the evolved weights to make up the ensemble. A large empirical study shows that, compared with some popular ensemble approaches such as Bagging and Boosting, GASEN can generate neural network ensembles with far smaller sizes but stronger generalization ability. Furthermore, in order to understand the working mechanism of GASEN, the bias-variance decomposition of the error is provided in this paper, which shows that the success of GASEN may lie in that it can significantly reduce the bias as well as the variance. <s> BIB010 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> We derive upper and lower limits on the majority vote accuracy with respect to individual accuracy p, the number of classiers in the pool (L), and the pairwise dependence between classiers, measured by Yule's Q statistic. Independence between individual classi- ers is typically viewed as an asset in classier fusion. We show that the majority vote with dependent classiers can potentially oer a dramatic improvement both over independent clas- siers and over the individual accuracy p. A functional relationship between the limits and the pairwise dependence Q is derived. Two patterns of the joint distribution for classier outputs (correct/incorrect) are identied to derive the limits: the pattern of success and the pattern of failure. The results support the intuition that negative pairwise dependence is benecial although not straightforwardly related to the accuracy. The pattern of success showed that for the highest improvement over p, all pairs of classiers in the pool should have the same negative dependence. <s> BIB011 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Abstract Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally. <s> BIB012 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Abstract The diversity of an ensemble of classifiers can be calculated in a variety of ways. Here a diversity metric and a means for altering the diversity of an ensemble, called “thinning”, are introduced. We evaluate thinning algorithms created by several techniques on 22 publicly available datasets. When compared to other methods, our percentage correct diversity measure shows a greatest correlation between the increase in voted ensemble accuracy and the diversity value. Also, the analysis of different ensemble creation methods indicates that they generate different levels of diversity. Finally, the methods proposed for thinning show that ensembles can be made smaller without loss in accuracy. <s> BIB013 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Ensembles are a widely used and effective technique in machine learning---their success is commonly attributed to the degree of disagreement, or 'diversity', within the ensemble. For ensembles where the individual estimators output crisp class labels, this 'diversity' is not well understood and remains an open research issue. For ensembles of regression estimators, the diversity can be exactly formulated in terms of the covariance between individual estimator outputs, and the optimum level is expressed in terms of a bias-variance-covariance trade-off. Despite this, most approaches to learning ensembles use heuristics to encourage the right degree of diversity. In this work we show how to explicitly control diversity through the error function. The first contribution of this paper is to show that by taking the combination mechanism for the ensemble into account we can derive an error function for each individual that balances ensemble diversity with individual accuracy. We show the relationship between this error function and an existing algorithm called negative correlation learning, which uses a heuristic penalty term added to the mean squared error function. It is demonstrated that these methods control the bias-variance-covariance trade-off systematically, and can be utilised with any estimator capable of minimising a quadratic error function, for example MLPs, or RBF networks. As a second contribution, we derive a strict upper bound on the coefficient of the penalty term, which holds for any estimator that can be cast in a generalised linear regression framework, with mild assumptions on the basis functions. Finally we present the results of an empirical study, showing significant improvements over simple ensemble learning, and finding that this technique is competitive with a variety of methods, including boosting, bagging, mixtures of experts, and Gaussian processes, on a number of tasks. <s> BIB014 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Ensemble approaches to classication and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error \diversity". However, the exact meaning of this concept is not clear from the literature, particularly for classication tasks. In this paper we rst review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classication literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classication error diversity. <s> BIB015 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> This paper presents a new approach for the cost-sensitive classification problems based on the Boosting ensemble of support vector machines (SVMs). Different from conventional Boosting ensemble learning methods that adjust the distribution of training instances for minimizing the misclassification rate, the presented approach adjusts the training data distribution so as to minimize the expected cost of classification. This approach has been applied successfully in Microcalcification (MC) detection which is a typical cost-sensitive classification problem in breast cancer diagnosis. Its performance is evaluated by means of Receiver Operating Characteristics (ROC) curves and the expected costs of classification. Experimental results have consistently confirmed that the ROC of the SVM ensemble classifier is very close to the curve enveloping the base classifier ROC curves. This characteristic illustrates that the SVM ensemble is able to always improve the performance of the classification. Furthermore, the experimental results demonstrate that the method presented is able to not only increase the area under the ROC curve (AUC) but also minimize the expected classification cost. <s> BIB016 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Diversity among the base classifiers is deemed to be important when constructing a classifier ensemble. Numerous algorithms have been proposed to construct a good classifier ensemble by seeking both the accuracy of the base classifiers and the diversity among them. However, there is no generally accepted definition of diversity, and measuring the diversity explicitly is very difficult. Although researchers have designed several experimental studies to compare different diversity measures, usually confusing results were observed. In this paper, we present a theoretical analysis on six existing diversity measures (namely disagreement measure, double fault measure, KW variance, inter-rater agreement, generalized diversity and measure of difficulty), show underlying relationships between them, and relate them to the concept of margin, which is more explicitly related to the success of ensemble learning algorithms. We illustrate why confusing experimental results were observed and show that the discussed diversity measures are naturally ineffective. Our analysis provides a deeper understanding of the concept of diversity, and hence can help design better ensemble learning algorithms. <s> BIB017 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> This paper studies the problem of pruning an ensemble of classifiers from a reinforcement learning perspective. It contributes a new pruning approach that uses the Q-learning algorithm in order to approximate an optimal policy of choosing whether to include or exclude each classifier from the ensemble. Extensive experimental comparisons of the proposed approach against state-of-the-art pruning and combination methods show very promising results. Additionally, we present an extension that allows the improvement of the solutions returned by the proposed approach over time, which is very useful in certain performance-critical domains. <s> BIB018 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> Although diversity in classifier ensembles is desirable, its relationship with the ensemble accuracy is not straightforward. Here we derive a decomposition of the majority vote error into three terms: average individual accuracy, “good” diversity and “bad diversity”. The good diversity term is taken out of the individual error whereas the bad diversity term is added to it. We relate the two diversity terms to the majority vote limits defined previously (the patterns of success and failure). A simulation study demonstrates how the proposed decomposition can be used to gain insights about majority vote classifier ensembles. <s> BIB019 </s> A survey of multiple classifier systems as hybrid systems <s> Diversity measures <s> The paper presents a cost-sensitive modification of the Adaptive Splitting and Selection (AdaSS) algorithm, which trains a combined classifier based on a feature space partitioning. In this study the algorithm considers constraints put on the cost of selected features, which are one of the key-problems in the clinical decision support systems. The modified version takes into consideration both the overall classification accuracy and the cost constraints, returning balanced solution for the problem at hand. Proposed method was evaluated on the basis of computer experiments run on cost-sensitive medical benchmark datasets. <s> BIB020
For regression problems, the variance of the outputs of ensemble members is a convenient diversity measure, because it was proved that the error of a compound model based on a weighted averaging of individual model outputs can be reduced according to increasing diversity BIB003 BIB004 . Brown et al. BIB014 showed a functional relation between diversity and individual regressor accuracy, allowing to control the bias-variance tradeoff systematically. For classification problems such theoretical results have not been proved yet, however many diversity measures have been proposed till now. On the one hand, it is intuitive that increasing diversity should lead to the better accuracy of the combined system, but there is no formal proof of this dependency BIB011 , as confirmed by the wide range of experimental results presented, e.g., in  . In BIB019 authors decomposed the error of the classification by majority voting into individual accuracy, good and bad diversities. The good diversity has positive impact on ensemble error reduction, whereas the bad diversity has the opposite effect. Sharkley et al. BIB006 proposed a hierarchy of four levels of diversity according to the answer of the majority rule, coincident failures, and possibility of at least one correct answer of ensemble members. Brown et al. BIB015 argue that this hierarchy is not appropriate when the ensemble diversity varies between feature subspaces. They formulated the following taxonomy of diversity measures: Pairwise measures averaging a measure between each classifier pair in an ensemble, such as Q-statistic BIB015 , kappa-statistics  , disagreement  and double-fault measure BIB011 BIB008 . Non-pairwise diversity measures comparing outputs of a given classifier and the entire ensemble, such as Kohavi-Wolpert variance  , a measure of inter-rater (inter-classifier) reliability BIB001 , the entropy measure BIB007 , the measure of difficulty BIB002 , generalized diversity BIB005 , and coincident failure diversity BIB009 . The analysis of several diversity measures BIB017 relating them to the concept of classifiers' margin, showed their limitations and the source of confusing empirical results. They relate the classifier selection to a NP-complete matrix cover problem, implying that ensemble design in fact a quite difficult combinatorial problem. Diversity measures usually employ the most valuable sub-ensemble in ensemble pruning processes  . To deal with the high computational complexity of ensemble pruning, several hybrid approaches have been proposed such as heuristic techniques BIB012 BIB013 , evolutionary algorithms BIB010  , reinforcement learning BIB018 , and competitive cross-validation techniques  . For classification tasks, the cost of acquiring feature values (which could be interpreted as the price for examination or time required to collect the data for decision making) can be critical. Some authors take it into consideration during the component classifier selection step BIB016 BIB020 . 