In this study, a new Stacked Generalization technique called Fuzzy Stacked Generalization (FSG) is proposed to minimize the difference between N -sample and large-sample classification error of the Nearest Neighbor classifier. The proposed FSG employs a new hierarchical distance learning strategy to minimize the error difference. For this purpose, we first construct an ensemble of base-layer fuzzy k- Nearest Neighbor (k-NN) classifiers, each of which receives a different feature set extracted from the same sample set. The fuzzy membership values computed at the decision space of each fuzzy k-NN classifier are concatenated to form the feature vectors of a fusion space. Finally, the feature vectors are fed to a meta-layer classifier to learn the degree of accuracy of the decisions of the base-layer classifiers for meta-layer classification. Rather than the power of the individual base layer-classifiers, diversity and cooperation of the classifiers become an important issue to improve the overall performance of the proposed FSG. A weak base-layer classifier may boost the overall performance more than a strong classifier, if it is capable of recognizing the samples, which are not recognized by the rest of the classifiers, in its own feature space. The experiments explore the type of the collaboration among the individual classifiers required for an improved performance of the suggested architecture. Experiments on multiple feature real-world datasets show that the proposed FSG performs better than the state of the art ensemble learning algorithms such as Adaboost, Random Subspace and Rotation Forest. On the other hand, compatible performances are observed in the experiments on single feature multi-attribute datasets.
In most of the experimental results given in the aforementioned studies, linear decision combination or aggregation method provides comparable or better performances than the other combination methods. However, performance evaluations of the stacked generalization methods reported in the literature are not consistent with each other. This fact is demonstrated by Dzeroski and Zenko in @cite_19 where they employ heterogeneous base-layer classifiers in their stacked generalization architecture. They report that their results contradict with the observations of the studies in the literature on SG. The contradictory results can be attributed to many non-linear relations among the parameters of the SG, such as the number and the structure of base-layer and meta-layer classifiers, and their feature, decision and fusion spaces.
@cite_19
1487121041 2162378706 2133935274 2023294425
Stacked Generalization (SG) is an ensemble learning technique, which aims to increase the performance of individual classifiers by combining them under a hierarchical architecture. In many applications, this technique performs better than the individual classifiers. However, in some applications, the performance of the technique goes astray, for the reasons that are not well-known. In this work, the performance of Stacked Generalization technique is analyzed with respect to the performance of the individual classifiers under the architecture. This work shows that the success of the SG highly depends on how the individual classifiers share to learn the training set, rather than the performance of the individual classifiers. The experiments explore the learning mechanisms of SG to achieve the high performance. The relationship between the performance of the individual classifiers and that of SG is also investigated. In this study, we introduce a new image classification technique using decision fusion. The proposed technique, called Meta-Fuzzified Yield Value (Meta-FYV), is based on two-layer Stacked Generalization (SG) architecture [1]. At the base-layer, the system, receives a set of feature vectors of various dimensions and dynamical ranges and outputs hypotheses through fuzzy transformations. Then, the hypotheses created by the base layer transformations are concatenated for building a regression equation at meta-layer. Experimental evidence indicates that the Meta-FYV is superior compared to one of the most successful Fuzzy SG methods, introduced by Akbas [2]. The main principle of stacked generalization is using a second-level generalizer to combine the outputs of base classifiers in an ensemble. In this paper, after presenting a short survey of the literature on stacked generalization, we propose to use regularized empirical risk minimization (RERM) as a framework for learning the weights of the combiner which generalizes earlier proposals and enables improved learning methods. Our main contribution is using group sparsity for regularization to facilitate classifier selection. In addition, we propose and analyze using the hinge loss instead of the conventional least squares loss. We performed experiments on three different ensemble setups with differing diversities on 13 real-world datasets of various applications. Results show the power of group sparse regularization over the conventional l"1 norm regularization. We are able to reduce the number of selected classifiers of the diverse ensemble without sacrificing accuracy. With the non-diverse ensembles, we even gain accuracy on average by using group sparse regularization. In addition, we show that the hinge loss outperforms the least squares loss which was used in previous studies of stacked generalization. We empirically evaluate several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. Among state-of-the-art stacking methods, stacking with probability distributions and multi-response linear regression performs best. We propose two extensions of this method, one using an extended set of meta-level features and the other using multi-response model trees to learn at the meta-level. We show that the latter extension performs better than existing stacking approaches and better than selecting the best classifier by cross validation.