Hoeffding Tree Algorithms for Anomaly Detection in Streaming Datasets: A Survey <s> Ensembles of Restricted Hoeffding Trees Using Stacking <s> This paper introduces stacked generalization, a scheme for minimizing the generalization error rate of one or more generalizers. Stacked generalization works by deducing the biases of the generalizer(s) with respect to a provided learning set. This deduction proceeds by generalizing in a second space whose inputs are (for example) the guesses of the original generalizers when taught with part of the learning set and trying to guess the rest of it, and whose output is (for example) the correct guess. When used with multiple generalizers, stacked generalization can be seen as a more sophisticated version of cross-validation, exploiting a strategy more sophisticated than cross-validation's crude winner-takes-all for combining the individual generalizers. When used with a single generalizer, stacked generalization is a scheme for estimating (and then correcting for) the error of a generalizer which has been trained on a particular learning set and then asked a particular question. After introducing stacked generalization and justifying its use, this paper presents two numerical experiments. The first demonstrates how stacked generalization improves upon a set of separate generalizers for the NETtalk task of translating text to phonemes. The second demonstrates how stacked generalization improves the performance of a single surface-fitter. With the other experimental evidence in the literature, the usual arguments supporting cross-validation, and the abstract justifications presented in this paper, the conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate. This paper ends by discussing some of the variations of stacked generalization, and how it touches on other fields like chaos theory. <s> BIB001 </s> Hoeffding Tree Algorithms for Anomaly Detection in Streaming Datasets: A Survey <s> Ensembles of Restricted Hoeffding Trees Using Stacking <s> The success of simple methods for classification shows that is is often not necessary to model complex attribute interactions to obtain good classification accuracy on practical problems. In this article, we propose to exploit this phenomenon in the data stream context by building an ensemble of Hoeffding trees that are each limited to a small subset of attributes. In this way, each tree is restricted to model interactions between attributes in its corresponding subset. Because it is not known a priori which attribute subsets are relevant for prediction, we build exhaustive ensembles that consider all possible attribute subsets of a given size. As the resulting Hoeffding trees are not all equally important, we weigh them in a suitable manner to obtain accurate classifications. This is done by combining the log-odds of their probability estimates using sigmoid perceptrons, with one perceptron per class. We propose a mechanism for setting the perceptrons’ learning rate using the change detection method for data streams, and also use to reset ensemble members (i.e., Hoeffding trees) when they no longer perform well. Our experiments show that the resulting ensemble classifier outperforms bagging for data streams in terms of accuracy when both are used in conjunction with adaptive naive Bayes Hoeffding trees, at the expense of runtime and memory consumption. We also show that our stacking method can improve the performance of a bagged ensemble. <s> BIB002
Bifet et al. in BIB002 , present an algorithm or classification model based on the ensemble of restricted decision trees, i.e. Hoeffding Trees, using Stacking. The authors indicate they use a stacking approach which is detailed on how each tree is built within this algorithm using attributes, log-odds of probabilities from predicted classes, applying sigmoid perceptrons, using perceptron classifiers and more. The authors also describe how their stacking approach differs from boosting and elaborate on the method they use for forming their ensemble classifier BIB001 The authors also note the significance of the use of Hoeffding Trees within their work since they are working with a data stream scenario. The authors thoroughly elaborate on the method they present in their proposed research work and why they use ADWIN, which they note is due to its theoretical guarantees on false positives within the context of change. The authors also elaborate on the benefits of ADWIN as well as how they use ADWIN for dealing with evolving data streams, replacing poorly performing ensemble members when an accuracy for one of the Hoeffding Trees has dropped significantly, and resetting the learning rate. The authors describe a detailed process of the ADWIN change detector to replace poorly performing ensemble members due to their decline in accuracy as well as the learning rate reset process. The authors in BIB002 Test-Then-Train or Prequential evaluation. The authors also note they performed a separate experiment for the disabling of the ADWIN-based change detection for resetting the learning rate and found a decrease in average accuracy in which they note these results reinforce the benefits of using ADWIN-based change detection for resetting the learning rate. The authors note from the experimental results it can be seen that the use of more trees with bagging using ADWIN is not any more accurate than the use of less trees with bagging using ADWIN, in which they believe the positive performance of their proposed algorithm is not related to the use of many trees. The authors believe the vital factor in the improvement of accuracy obtained is their methodology of using attribute subsets and perceptron weighting. The authors note during the experimental evaluation, they found an increase in ensemble diversity produced from their new stacking method in which they indicate is due the tight clusteration of the ADWIN bagging ensembles and increased discrepancy amdist classifiers within the ensemble produced by the stacking method. To expose this divergence, the authors use the Kappa statistic method k  and describe its use within their newly stacking based classifier to increase ensemble diversity. The authors also introduce a new strategy to replace ensemble classifiers. We have organized some of the evaluation metrics produced from the authors' experimental evaluation based on the results illustrated within BIB002 , these evaluation metrics are included in a comparison chart provided within Table 1 in Section 6. Advantages of this research: The authors in BIB002 ing cyber data sets. Since an ensemble of classifiers is stronger than a single classifier, the method and techniques presented in this proposed research work can be applied to effectively address problems in the domain of anomaly detection and cyber data streams. This idea is also explained more in detail in Section 6. Disadvantages of this research: The authors note some deficiencies with initial experiments of pruning the ensembles and discarding the least important ensemble member, in which their goal is to address this with smarter pruning methods and techniques to increase accuracy classification.