A survey of multiple classifier systems as hybrid systems <s> Introduction <s> This study explores the scope for achieving enhanced recognition system performance through deployment of a composite classifier system consisting of two or more component classifiers which belong to different categories. The domains of deployment of these individual components (classifiers) are determined by optimal partitioning of the problem space. The criterion for such optimal partitioning is determined in each case by the characteristics of the classifier components. An example, in terms of partitioning the feature space for optimal deployment of a composite system consisting of the linear and nearest neighbor (NN) classifiers as its components, is presented to illustrate the concepts, the associated methodology, and the possible benefits one could expect through such composite classifier system design. Here, the optimality of the partitioning is dictated by the linear class separability limitation of the linear classifier and the computational demand characteristics of the NN classifier. Accordingly, the criterion for the optimal feature space partitioning is set to be the minimization of the domain of application of the NN classifier, subject to the constraint that the linear classifier is to be deployed only in regions satisfying the underlying assumption of linear separability of classes. While many alternatives are available for the solution of the resulting constrained optimization problem, a specific technique-Sequential Weight Increasing Factor Technique (SWIFT)- was employed here for convenience in view of previous successful experience with this technique in other application areas. Numerical results derived using the well-known IRIS data set are furnished to demonstrate the effectiveness of the new concepts and methodology. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Consider a group of people confronted with a dichotomous choice (for example, a yes or no decision). Assume that we can characterize each person by a probability, pi, of making the 'better' of the two choices open to the group, such that we define 'better' in terms of some linear ordering of the alternatives. If individual choices are independent, and if the a priori likelihood that either of the two choices is correct is one half, we show that the group decision procedure that maximizes the likelihood that the group will make the better of the two choices open to it is a weighted voting rule that assigns weights, wi, such that <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> We consider the problem of using queries to learn an unknown concept. Several types of queries are described and studied: membership, equivalence, subset, superset, disjointness, and exhaustiveness queries. Examples are given of efficient learning methods using various subsets of these queries for formal domains, including the regular languages, restricted classes of context-free languages, the pattern languages, and restricted types of prepositional formulas. Some general lower bound techniques are given. Equivalence queries are compared with Valiant's criterion of probably approximately correct identification under random sampling. <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Abstract Considerable literature has accumulated over the years regarding the combination of forecasts. The primary conclusion of this line of research is that forecast accuracy can be substantially improved through the combination of multiple individual forecasts. Furthermore, simple combination methods often work reasonably well relative to more complex combinations. This paper provides a review and annotated bibliography of that literature, including contributions from the forecasting, psychology, statistics, and management science literatures. The objectives are to provide a guide to the literature for students and researchers and to help researchers locate contributions in specific areas, both theoretical and applied. Suggestions for future research directions include (1) examination of simple combining approaches to determine reasons for their robustness, (2) development of alternative uses of multiple forecasts in order to make better use of the information they contain, (3) use of combined forecasts as benchmarks for forecast evaluation, and (4) study of subjective combination procedures. Finally, combining forecasts should become part of the mainstream of forecasting practice. In order to achieve this, practitioners should be encouraged to combine forecasts, and software to produce combined forecasts easily should be made available. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. > <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Possible solutions to the problem of combining classifiers can be divided into three categories according to the levels of information available from the various classifiers. Four approaches based on different methodologies are proposed for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-nearest-neighbor, and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly. For example, on the US zipcode database, 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as high reliability with 95% recognition, 0% substitution, and 5% rejection. > <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> A multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures. Decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem. The rankings can be combined by methods that either reduce or rerank a given set of classes. An intersection method and union method are proposed for class set reduction. Three methods based on the highest rank, the Borda count, and logistic regression are proposed for class set reranking. These methods have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness. > <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> The probably approximately correct learning model Occam's razor the Vapnik-Chervonenkis dimension weak and strong learning learning in the presence of noise inherent unpredictability reducibility in PAC learning learning finite automata by experimentation appendix - some tools for probabilistic analysis. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Multiple expert decision combination has received much attention in recent years. This is a multi-disciplinary branch of pattern recognition which has extensive applications in numerous fields including robotic vision, artificial intelligence, document processing, office automation, human-computer interfaces, data acquisition, storage and retrieval, etc. In recent years, this application area has been extended to forensic science, including the identification of individuals using measures depending on biometrics, security and other applications. In this paper, a generalised multi-expert multi-level decision combination strategy, the serial combination approach, has been investigated from the dual viewpoints of theoretical analysis and practical implementation. Different researchers have implicitly utilised various approaches based on this concept over the years in a wide spectrum of application domains, but a comprehensive, coherent and generalised presentation of this approach from both theoretical and implementation viewpoints has not been attempted. While presenting here a unified framework for serial multiple expert decision combination, it is shown that many multi-expert approaches reported in the literature can be easily represented within the proposed framework. Detailed theoretical and practical discussions of the various performance results with these combinations, analysis of the internal processing of this approach, a case study for testing the theoretical framework, issues relating to processing overheads associated with the implementation of this approach, general comments on its applicability to various task domains and the generality of the approach in terms of reevaluating previous research have also been incorporated. <s> BIB010 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field. <s> BIB011 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly. <s> BIB012 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> A fruitful direction for future data mining research will be the development of techniques that incorporate privacy concerns. Specifically, we address the following question. Since the primary task in data mining is the development of models about aggregated data, can we develop accurate models without access to precise information in individual data records? We consider the concrete case of building a decision-tree classifier from training data in which the values of individual records have been perturbed. The resulting data records look very different from the original records and the distribution of data values is also very different from the original distribution. While it is not possible to accurately estimate original values in individual data records, we propose a novel reconstruction procedure to accurately estimate the distribution of original data values. By using these reconstructed distributions, we are able to build classifiers whose accuracy is comparable to the accuracy of classifiers built with the original data. <s> BIB013 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems. <s> BIB014 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> This paper reviews the supervised learning versions of the no-free-lunch theorems in a simplified form. It also discusses the significance of those theorems, and their relation to other aspects of supervised learning. <s> BIB015 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> It is widely acknowledged that face recognition could play an important role in advanced video-based surveillance systems, mainly because it is non intrusive and does not require people cooperation [1, 2]. Unfortunately, face recognition algorithms showed to suffer a lot from the high variability of environmental conditions. As an example, the effectiveness of face recognition strongly depends on lighting conditions and on variations in the subject’s pose and expression in front of the camera. This obviously limits their application to real video-surveillance tasks. On the other hand, face is considered a very good biometric. People recognize each other through the face, the acquisition process is non-intrusive, and does not require the collaboration of the subject to be recognized. Therefore, face recognition is a very active research field with many applications. For the purposes of this chapter, the face recognition applications can be subdivided in two types: applications in controlled and uncontrolled environments. One of the main applications of the first type is the so called “identity authentication”. A person submits to the automatic identity verification system its face (frontal and/or profile view) and declares her/his identity. The system matches the acquired face with the “template” stored in its data base, and classifies the person as a “genuine” (i.e., the claimed identity is accepted) or an “impostor”. <s> BIB016 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> In this paper, we investigate the usefulness of the reject option in text categorisation systems. The reject option is introduced by allowing a text classifier to withhold the decision of assigning or not a document to any subset of categories, for which the decision is considered not sufficiently reliable. To automatically handle rejections, a two-stage classifier architecture is used, in which documents rejected at the first stage are automatically classified at the second stage, so that no rejections eventually remain. The performance improvement achievable by using the reject option is assessed on a real text categorisation task, using the well known Reuters data set. <s> BIB017 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Systems for inducing concept descriptions from examples are valuable tools for assisting in the task of knowledge acquisition for expert systems. This paper presents a description and empirical evaluation of a new induction system. CN2, designed for the efficient induction of simple, comprehensible production rules in domains where problems of poor description language and/or noise may be present. Implementations of the CN2, ID3, and AQ algorithms are compared on three medical classification tasks. <s> BIB018 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy. <s> BIB019 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> This paper introduces a new representation for Boolean functions, called decision lists, and shows that they are efficiently learnable from examples. More precisely, this result is established for k-;DL – the set of decision lists with conjunctive clauses of size k at each decision. Since k-DL properly includes other well-known techniques for representing Boolean functions such as k-CNF (formulae in conjunctive normal form with at most k literals per clause), k-DNF (formulae in disjunctive normal form with at most k literals per term), and decision trees of depth k, our result strictly increases the set of functions that are known to be polynomially learnable, in the sense of Valiant (1984). Our proof is constructive: we present an algorithm that can efficiently construct an element of k-DL consistent with a given set of examples, if one exists. <s> BIB020 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> This book, which is wholly devoted to the subject of model combination, is divided into ten chapters. In addition to the first two introductory chapters, the book covers some of the following topics: multiple classifier systems; combination methods when the base classifier outputs are 0/1; methods when the outputs are continuous, e.g., posterior probabilities; methods for classifier selection; bagging and boosting; the theory of fixed combination rules; and the concept of diversity. Overall, it is a very well-written monograph. It explains and analyzes different approaches comparatively so that the reader can see how they are similar and how they differ. The literature survey is extensive. The MATLAB code for many methods is given in chapter appendices allowing readers to play with the explained methods or apply them quickly to their own data. The book is a must-read for researchers and practitioners alike. <s> BIB021 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Broad classes of statistical classification algorithms have been developed and applied successfully to a wide range of real-world domains. In general, ensuring that the particular classification algorithm matches the properties of the data is crucial in providing results that meet the needs of the particular application domain. One way in which the impact of this algorithm/application match can be alleviated is by using ensembles of classifiers, where a variety of classifiers (either different types of classifiers or different instantiations of the same classifier) are pooled before a final classification decision is made. Intuitively, classifier ensembles allow the different needs of a difficult problem to be handled by classifiers suited to those particular needs. Mathematically, classifier ensembles provide an extra degree of freedom in the classical bias/variance tradeoff, allowing solutions that would be difficult (if not impossible) to reach with only a single classifier. Because of these advantages, classifier ensembles have been applied to many difficult real-world problems. In this paper, we survey select applications of ensemble methods to problems that have historically been most representative of the difficulties in classification. In particular, we survey applications of ensemble methods to remote sensing, person recognition, one vs. all recognition, and medicine. <s> BIB022 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Ensemble methodology, which builds a classification model by integrating multiple classifiers, can be used for improving prediction performance. Researchers from various disciplines such as statistics, pattern recognition, and machine learning have seriously explored the use of ensemble methodology. This paper presents an updated survey of ensemble methods in classification tasks, while introducing a new taxonomy for characterizing them. The new taxonomy, presented from the algorithm designer's point of view, is based on five dimensions: inducer, combiner, diversity, size, and members' dependency. We also propose several selection criteria, presented from the practitioner's point of view, for choosing the most suitable ensemble method. <s> BIB023 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Ensemble methods have been called the most influential development in Data Mining and Machine Learning in the past decade. They combine multiple models into one usually more accurate than the best of its components. Ensembles can provide a critical boost to industrial challenges -- from investment timing to drug discovery, and fraud detection to recommendation systems -- where predictive accuracy is more vital than model interpretability. Ensembles are useful with all modeling algorithms, but this book focuses on decision trees to explain them most clearly. After describing trees and their strengths and weaknesses, the authors provide an overview of regularization -- today understood to be a key reason for the superior performance of modern ensembling algorithms. The book continues with a clear description of two recent developments: Importance Sampling (IS) and Rule Ensembles (RE). IS reveals classic ensemble methods -- bagging, random forests, and boosting -- to be special cases of a single algorithm, thereby showing how to improve their accuracy and speed. REs are linear rule models derived from decision tree ensembles. They are the most interpretable version of ensembles, which is essential to applications such as credit scoring and fault diagnosis. Lastly, the authors explain the paradox of how ensembles achieve greater accuracy on new data despite their (apparently much greater) complexity.This book is aimed at novice and advanced analytic researchers and practitioners -- especially in Engineering, Statistics, and Computer Science. Those with little exposure to ensembles will learn why and how to employ this breakthrough method, and advanced practitioners will gain insight into building even more powerful models. Throughout, snippets of code in R are provided to illustrate the algorithms described and to encourage the reader to try the techniques. (edited by author) <s> BIB024 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> Our discussion in this article centers around various issues related to the use of anycasting in connection-oriented computer networks. Anycast is defined as a one-to-one-of-many transmission to deliver a packet to one of many hosts. Anycasting can be applied if the same content is replicated over many locations in the network. Examples of network techniques that apply anycasting are Content Delivery Networks (CDNs), Domain Name Service (DNS), Peer-to-Peer (P2P) systems. The role of anycasting is growing concurrently with the popularity of electronic music, movies, and other content required by Internet users. In this work we focus on the optimization of anycast flows in connection-oriented networks. We formulate a model of anycast connections and next propose a heuristic algorithm based on the Lagrangean relaxation aimed to optimize jointly routes for anycast and unicast connections. Results of numerical experiments are presented and evaluated. Finally, we analyze briefly problems related to anycasting in dynamic routing and multi-layer networks. <s> BIB025 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> The application of a “committee of experts” or ensemble learning to artificial neural networks that apply unsupervised learning techniques is widely considered to enhance the effectiveness of such networks greatly. This book examines in one of its chapters the potential of the ensemble meta-algorithm by describing and testing a technique based on the combination of ensembles and statistical PCA that is able to determine the presence of outliers in high-dimensional data sets and to minimize outlier effects in the final results. After that, it presents its central contribution, which consists on an algorithm for the ensemble fusion of topology-preserving maps, referred to as Weighted Voting Superposition (WeVoS), which has been devised to improve data exploration by 2-D visualization over multi-dimensional data sets. This generic algorithm is applied in combination with several other models taken from the family of topology preserving maps, such as the SOM, ViSOM, SIM and Max-SIM. A range of quality measures for topology preserving maps that are proposed in the literature are used to validate and compare WeVoS with other algorithms. The experimental results demonstrate that, in the majority of cases, the WeVoS algorithm outperforms earlier map-fusion methods and the simpler versions of the algorithm with which it is compared. All the algorithms are tested in different artificial data sets and in several of the most common machine-learning data sets in order to corroborate their theoretical properties. Moreover, a real-life case-study taken from the food industry demonstrates the practical benefits of their applications to more complex problems. <s> BIB026 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> More recently, neural network techniques and fuzzy logic inference systems have been receiving an increasing attention. At the same time, methods of establishing decision by a group of classifiers are regarded as a general problem in various application areas of pattern recognition. Similarly to standard two-class pattern recognition methods, one-class classifiers hardly ever fit the data distribution perfectly. The paper presents fuzzy models for one-class classifier combination, compares their computational and expected space complexity with the one from ECOC and decision templates, presents possibility to speed up a fuser processing by means of multithreading. <s> BIB027 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> We present a two stage sequential ensemble where data samples whose output from the first classifier fall in a low confidence output interval (LCOI) are processed by a second stage classifier. Training is composed of three processes: training the first classifier, determining the LCOI of the first classifier, and training the second classifier upon the data items whose output fall in the LCOI. The LCOI is determined varying a threshold on the false positive rate (FPR) and false negative rate (FNR) curves. We have tested the approach on a database of feature vectors for the classification of Alzheimer's disease (AD) and control subjects extracted from structural magnetic resonance imaging (sMRI) data. In this paper, we focus on the combinations obtained when the first classifier is a relevance vector machine (RVM). Obtained results improve over previous results for this database. <s> BIB028 </s> A survey of multiple classifier systems as hybrid systems <s> Introduction <s> This chapter introduces the foundations of machine learning that are necessary to motivate its combination to evolutionary algorithms and image processing. In the chapter, we will describe several machine learning tools, suitable for a wide variety of data and tasks. The goal is not to describe the total number of techniques, but instead to present a general view of the field through the description of the mail problems considered in machine learning. <s> BIB029
Hybrid Intelligent Systems offer many alternatives for unorthodox handling of realistic increasingly complex problems, involving ambiguity, uncertainty and high-dimensionality of data. They allow to use both a priori knowledge and raw data to compose innovative solutions. Therefore, there is growing attention to this multidisciplinary research field in the computer engineering research community. Hybridization appears in many domains of human activity. It has an immediate natural inspiration in the human biological systems, such as the Central Nervous System, which is a de facto hybrid composition of many diverse computational units, as discussed since the early days of computer science, e.g., by von Neumann  or Newell  . Hybrid approaches seek to exploit the strengths of the individual components, obtaining enhanced performance by their combination. The famous ''no free lunch'' theorem BIB015 stated by Wolpert may be extrapolated to the point of saying that there is no single computational view that solves all problems. Fig. 1 is a rough representation of the computational domains covered by the Hybrid Intelligent System approach. Some of them deal with the uncertainty and ambiguity in the data by probabilistic or fuzzy representations and feature extraction. Others deal with optimization problems appearing in many facets of the intelligent system design and problem solving, either following a nature inspired or a stochastic process approach. Finally, classifiers implementing the intelligent decision process are also subject to hybridization by various forms of combination. In this paper, we focus in this specific domain, which is in an extraordinary effervescence nowadays, under the heading of Multi-Classifier Systems (MCS). Referring to classification problems, Wolpert's theorem has an specific lecture: there is not a single classifier modeling approach which is optimal for all pattern recognition tasks, since each has its own domain of competence. For a given classification task, we expect the MCS to exploit the strengths of the individual classifier models at our disposal to produce the high quality compound recognition system overcoming the performance of individual classifiers. Summarizing: Hybrid Intelligent Systems (HIS) are free combinations of computational intelligence techniques to solve a given problem, covering al computational phases from data normalization up to final decision making. Specifically, they mix heterogeneous fundamental views blending them into one effective working system. Information Fusion covers the ways to combine information sources in a view providing new properties that may allow to solve better or more efficiently the proposed problem. Information sources can be the result of additional computational processes. Multi-Classifier Systems (MCS) focus on the combination of classifiers form heterogenous or homogeneous modeling backgrounds to give the final decision. MCS are therefore a subcategory of HIS.  of independent binary classifiers with appropriately defined weights. In 1979 Dasarathy and Sheela combined a linear classifier and one k-NN classifier BIB001 , suggesting to identify the region of the feature space where the classifiers disagree. The k-NN classifier gives the answer of the MCS for the objects coming from the conflictive region and by the linear one for the remaining objects. Such strategy significantly decreases the exploitation cost of whole classifier system. This was the first work introducing a classifier selection concept, however the same idea was developed independently in 1981 by Rastrigin and Erenstein  performing first a feature space partitioning and, second, assigning to each partition region an individual classifier that achieves the best classification accuracy over it. Other early relevant works formulated conclusions regarding MCS 's classification quality, such as BIB005 who considered a neural network ensemble, BIB006 with majority voting applied to handwriting recognition, Turner in 1996  showed that averaging outputs of an infinite number of unbiased and independent classifiers can lead to the same response as the optimal Bayes classifier, Ho BIB008 underlined that a decision combination function must receive useful representation of each classifier's decision. Specifically, they considered several method based on decision ranks, such as Borda count. Finally, the landmark works devoted introducing bagging BIB019 and boosting   which are able to produce strong classifiers BIB009 , in the (Probably Approximately Correct) theory BIB003 sense, on the basis of the weak one. Nowadays MCS, are highlighted by review articles as a hot topic and promising trend in pattern recognition BIB011 BIB022   BIB023 . These reviews include the books by Kuncheva BIB021 , Rokach  , Seni and Edler BIB024 , and Baruque and Corchado BIB026 . Even leading-edge general machine learning handbooks such as  BIB029  include extensive presentations of MCS concepts and architectures. The popularity of this approach is confirmed by the growing trend in the number of publications shown in Fig. 2 . The figure reproduces the evolution of the number of references retrieved by the application of specific keywords related to MCS since 1990. The experiment was repeated on three well known academic search sites. The growth in the number of publications has an exponential trend. The last entry of the  We can retrace decision combination long way back in history. Perhaps the first worthy reference is the Greek democracy (meaning government of the people) ruling that full citizens have an equal say in any decision that affects their life. Greeks believed in the community wisdom, meaning that the rule of the majority will produce the optimal joint decision. In 1785 Condorcet formulated the Jury Theorem about the misclassification probability of a group of independent voters BIB002 ], providing the first result measuring the quality of classifier committee. Fig. 2 . Evolution of the number of publications per year ranges retrieved from the keywords specified in the plot legend. Each plot corresponds to searching site: the top to Google Scholar; the center to the Web of Knowledge, the bottom to Scopus. The first entry of the plots is for publications prior to 1990. The last entry is only for the last 2 years. plots corresponds to the last 2 years, and some of the keywords give as many references as in the previous 5 years. Advantages. Dietterich BIB012 summarized the benefits of MCS: (a) allowing to filter out hypothesis that, though accurate, might be incorrect due to a small training set, (b) combining classifiers trained starting from different initial conditions could overcome the local optima problem, and (c) the true function may be impossible to be modeled by any single hypothesis, but combinations of hypotheses may expand the space of representable functions. Rephrasing it, there is widespread acknowledgment of the following advantages of MCS: MCS behave well in the two extreme cases of data availability: when we have very scarce data samples for learning, and when we have a huge amount of them at our disposal. In the scarcity case, MCS can exploit bootstrapping methods, such as bagging or boosting. Intuitive reasoning justifies that the worst classifier would be out of the selection by this method BIB016 , e.g., by individual classifier output averaging BIB007 . In the event of availability of a huge amount of learning data samples, MCS allow to train classifiers on dataset's partitions and merge their decision using appropriate combination rule  . Combined classifier can outperform the best individual classifier BIB004 . Under some conditions (e.g., majority voting by a group of independent classifiers) this improvement has been proven analytically  . Many machine learning algorithms are de facto heuristic search algorithms. For example the popular decision tree induction method C4.5  uses a greedy search approach, choosing the search direction according to an heuristic attribute evaluation function. Such an approach does not assure an optimal solution. Thus, the combined algorithm, which could start its work from different initial points of the search space, is equivalent to a multi-start local random search which increases the probability of finding an optimal model. MCS can easily be implemented in efficient computing environments such as parallel and multithreaded computer architectures BIB027 . Another attractive area of implementation solutions is distributed computing systems (i.e.: P2P, Grid or Cloud computing)  BIB025 , especially when a database is partitioned for privacy reasons BIB013 so that partial solutions must be computed on each partition and only the final decision is available as the combination of the networked decision. Wolpert stated that each classifier has its specific competence domain BIB015 , where they overcome other competing algorithms, thus it is not possible to design a single classifier which outperforms another ones for each classification tasks. MCS try to select always the local optimal model from the available pool of trained classifiers. System structure. The general structure of MCS is depicted in Fig. 3 following a classical pattern recognition BIB014 application structure. The most informative or discriminant features describing the objects are input to the classifier ensemble, formed by a set of complementary and diverse classifiers. An appropriate fusion method combines the individual classifier outputs optimally to provide the system decision. According to Ho  , two main MCS design approaches can be distinguished. On one hand, the so-called coverage optimization approach tries to cover the space of possible models by the generation of a set of mutually complementary classifiers whose combination provides optimal accuracy. On the other hand, the so-called decision optimization approach concentrates on designing and training an appropriate decision combination function over a set of individual classifier given in advance  .The main issues in MCS design are: System topology: How to interconnect individual classifiers. Ensemble design: How to drive the generation and selection of a pool of valuable classifiers. Fuser design: How to build a decision combination function (fuser) which can exploit the strengths of the selected classifiers and combine them optimally. Fig. 4 illustrates the two canonical topologies employed in MCS design. The overwhelming majority of MCS reported in the literature is structured in a parallel topology BIB021 . In this architecture, each classifier is feed the same input data, so that the final decision of the combined classifier output is made on the basis of the outputs of the individual classifiers obtained independently. Alternatively, in the serial (or conditional) topology, individual classifiers are applied in sequence, implying some kind of ranking or ordering over them. When the primary classifier cannot be trusted to classify a given object, e.g., because of the low support/confidence in its result, then the data is feed to a secondary classifier  BIB010 , and so on, adding classifiers in sequence. This topology is adequate when the cost of classifier exploitation is important, so that the primary classifier is the computationally cheapest one, and secondary classifiers have higher exploitation cost BIB017 . This model can be applied to classifiers with the so-called reject option as well  . In BIB028 the first classifier in the pipeline gives an estimation of the certainty of the classification, so that uncertain data samples are sent to a second classifier, specialized in difficult instances. We notice the similarity of such approach to the ordered set of rules BIB018 or decision list BIB020 , when we consider each rule as the classifier.