A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k > 2 values (i.e., k "classes"). The definition is acquired by studying collections of training examples of the form (xi, f(xi)). Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision-tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error-correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision-tree pruning. Finally, we show that--like the other methods--the error-correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> In the problem of one-class classification target objects should be distinguished from outlier objects. In this problem it is assumed that only information of the target class is available while nothing is known about the outlier class. Like standard two-class classifiers, one-class classifiers hardly ever fit the data distribution perfectly. Using only the best classifier and discarding the classifiers with poorer performance might waste valuable information. To improve performance the results of different classifiers (which may differ in complexity or training algorithm) can be combined. This can not only increase the performance but it can also increase the robustness of the classification. Because for one-class classifiers only information of one of the classes is present, combining one-class classifiers is more difficult. In this paper we investigate if and how one-class classifiers can be combined best in a handwritten digit recognition problem. <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> The generalization from two-class classification to multiclass classification is not straightforward for discriminants which are not based on density estimation. Simple combining methods use voting, but this has the drawback of inconsequent labelings and ties. More advanced methods map the discriminant outputs to approximate posterior probability estimates and combine these, while other methods use error-correcting output codes. In this paper we want to show the possibilities of simple generalizations of the two-class classification, using voting and combinations of approximate posterior probabilities. <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> In this paper, we propose a multi-category classification method that combines binary classifiers through soft-max function. Posteriori probabilities are also obtained. Both, one-versus-all and one-versus-one classifiers can be used in the combination. Empirical comparison shows that the proposed method is competitive with other implementations of one-versus-all and one-versus-one methods in terms of both classification accuracy and posteriori probability estimate. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> We study the problem of multiclass classification within the framework of error correcting output codes (ECOC) using margin-based binary classifiers. Specifically, we address two important open problems in this context: decoding and model selection. The decoding problem concerns how to map the outputs of the classifiers into class codewords. In this paper we introduce a new decoding function that combines the margins through an estimate of their class conditional probabilities. Concerning model selection, we present new theoretical results bounding the leave-one-out (LOO) error of ECOC of kernel machines, which can be used to tune kernel hyperparameters. We report experiments using support vector machines as the base binary classifiers, showing the advantage of the proposed decoding function over other functions of I he margin commonly used in practice. Moreover, our empirical evaluations on model selection indicate that the bound leads to good estimates of kernel parameters. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> Pairwise coupling is a popular multi-class classification method that combines together all pairwise comparisons for each pair of classes. This paper presents two approaches for obtaining class probabilities. Both methods can be reduced to linear systems and are easy to implement. We show conceptually and experimentally that the proposed approaches are more stable than two existing popular methods: voting and [3]. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> Since the early days of research on intrusion detection, anomaly-based approaches have been proposed to detect intrusion attempts. Attacks are detected as anomalies when compared to a model of normal (legitimate) events. Anomaly-based approaches typically produce a relatively large number of false alarms compared to signature-based IDS. However, anomaly-based IDS are able to detect never-before-seen attacks. As new types of attacks are generated at an increasing pace and the process of signature generation is slow, it turns out that signature-based IDS can be easily evaded by new attacks. The ability of anomaly-based IDS to detect attacks never observed in the wild has stirred up a renewed interest in anomaly detection. In particular, recent work focused on unsupervised or unlabeled anomaly detection, due to the fact that it is very hard and expensive to obtain a labeled dataset containing only pure normal events. The unlabeled approaches proposed so far for network IDS focused on modeling the normal network traffic considered as a whole. As network traffic related to different protocols or services exhibits different characteristics, this paper proposes an unlabeled Network Anomaly IDS based on a modular Multiple Classifier System (MCS). Each module is designed to model a particular group of similar protocols or network services. The use of a modular MCS allows the designer to choose a different model and decision threshold for different (groups of) network services. This also allows the designer to tune the false alarm rate and detection rate produced by each module to optimize the overall performance of the ensemble. Experimental results on the KDD-Cup 1999 dataset show that the proposed anomaly IDS achieves high attack detection rate and low false alarm rate at the same time. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> Weighted voting is the commonly used strategy for combining predictions in pairwise classification. Even though it shows good classification performance in practice, it is often criticized for lacking a sound theoretical justification. In this paper, we study the problem of combining predictions within a formal framework of label ranking and, under some model assumptions, derive a generalized voting strategy in which predictions are properly adapted according to the strengths of the corresponding base classifiers. We call this strategy adaptive voting and show that it is optimal in the sense of yielding a MAP prediction of the class label of a test instance. Moreover, we offer a theoretical justification for weighted voting by showing that it yields a good approximation of the optimal adaptive voting prediction. This result is further corroborated by empirical evidence from experiments with real and synthetic data sets showing that, even though adaptive voting is sometimes able to achieve consistent improvements, weighted voting is in general quite competitive, all the more in cases where the aforementioned model assumptions underlying adaptive voting are not met. In this sense, weighted voting appears to be a more robust aggregation strategy. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Diversifying outputs <s> The paper shows the possibilities of generalizing the two-class classification into multi-class classification by means of a fuzzy inference system. Fuzzy combiner harnesses the support values from classifiers to provide final response having no other restrictions on their structure. We compare proposed combination methods with ECOC and two variations of decision templates, based on Euclidean and symmetric distance. The effectiveness of the proposed combination method based on the fuzzy logic theory is also evaluated via computer experiments carried out on benchmark datasets. <s> BIB009
MCS diversity can be enforced by the manipulation of the individual classifier outputs, so that an individual classifier is designed to classify only some classes in the problem. The combination method should restore the whole class label set, e.g., a multi-class classification problem can be decomposed into a set of binary classification problems BIB003 BIB001 . The most popular propositions of two-class classifier combinations are: OAO (one-against-one) and OAA (one-against-all) BIB004 , where at least one predictor relates to each class. The model that a given object belongs to a chosen class is tested against the alternative of the feature vector belonging to any other class. In the OAA method, a classifier is trained to separate a chosen class from the remaining ones. OAA returns class with maximum support. In more general approaches, the combination of individual outputs is made by finding the closest class, in some sense, to the code given by the outputs of the individual classifiers. ECOC (Error Correcting Output Codes) model was proposed by Dieterich and Bakiri BIB001 , who assumed that a set of classifiers produces sequence of bits which is related to code-words during training. The ECOC points at the class with the smallest Hamming distance to its codeword. Passerini et al. showed advantages of this method over traditional ones for the ensemble of support vector machines BIB005 . Recently several interesting propositions on how to combine the binary classifiers were proposed. Wu et al. BIB006 used pairwise coupling, Friedman employed Max-Win rule  , Hüllermeier proposed the adaptive weighted voting procedure BIB008 . A comprehensive recent survey of binary classifier ensembles is  . It worth mentioning the one-class classification model which is the special case of binary classifier trained in the absence of counterexamples. Its main goal is to model normality in order to detect anomaly or outliers from the target class  . To combine such classifiers the typical methods developed for binary ones are used BIB002 but it is worth mention the work by Wilk and Wozniak where authors restored multi-class classification task using a pool of one-class classifiers and the fuzzy inference system BIB009 . The combination methods dedicated the one-class classifiers still await a proper attention BIB007 .