A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Ensemble methods have recently garnered a great deal of attention in the machine learning community. Techniques such as Boosting and Bagging have proven to be highly effective but require repeated resampling of the training data, making them inappropriate in a data mining context. The methods presented in this paper take advantage of plentiful data, building separate classifiers on sequential chunks of training points. These classifiers are combined into a fixed-size ensemble using a heuristic replacement strategy. The result is a fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift. <s> BIB001 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Many of today's real world domains require online classification tasks in very demanding situations. This work presents the results of applying the CD3 algorithm to telecommunications call data. CD3 enables the detection of concept drift in the presence of noise within real time data. The application detects the drift using a TSAR methodology and applies a purging mechanism as a corrective action. The main focus of this work is to identify from customer files and call records if the profile of customers registering for a 'friends and family' service is changing over a period of time. We will begin with a review of the CD3 application and the presentation of the data. This will conclude with experimental results. <s> BIB002 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Recently, mining data streams with concept drifts for actionable insights has become an important and challenging task for a wide range of applications including credit card fraud protection, target marketing, network intrusion detection, etc. Conventional knowledge discovery tools are facing two challenges, the overwhelming volume of the streaming data, and the concept drifts. In this paper, we propose a general framework for mining concept-drifting data streams using weighted ensemble classifiers. We train an ensemble of classification models, such as C4.5, RIPPER, naive Beyesian, etc., from sequential chunks of the data stream. The classifiers in the ensemble are judiciously weighted based on their expected classification accuracy on the test data under the time-evolving environment. Thus, the ensemble approach improves both the efficiency in learning the model and the accuracy in performing classification. Our empirical study shows that the proposed methods have substantial advantage over single-classifier approaches in prediction accuracy, and the ensemble framework is effective for a variety of classification models. <s> BIB003 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Algorithms for tracking concept drift are important for many applications. We present a general method based on the weighted majority algorithm for using any online learner for concept drift. Dynamic weighted majority (DWM) maintains an ensemble of base learners, predicts using a weighted-majority vote of these "experts", and dynamically creates and deletes experts in response to changes in performance. We empirically evaluated two experimental systems based on the method using incremental naive Bayes and incremental tree inducer [ITI] as experts. For the sake of comparison, we also included Blum's implementation of weighted majority. On the STAGGER concepts and on the SEA concepts, results suggest that the ensemble method learns drifting concepts almost as well as the base algorithms learn each concept individually. Indeed, we report the best overall results for these problems to date. <s> BIB004 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics. <s> BIB005 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> The recent advances in hardware and software have enabled the capture of different measurements of data in a wide range of fields. These measurements are generated continuously and in a very high fluctuating data rates. Examples include sensor networks, web logs, and computer network traffic. The storage, querying and mining of such data sets are highly computationally challenging tasks. Mining data streams is concerned with extracting knowledge structures represented in models and patterns in non stopping streams of information. The research in data stream mining has gained a high attraction due to the importance of its applications and the increasing generation of streaming information. Applications of data stream analysis can vary from critical scientific and astronomical applications to important business and financial ones. Algorithms, systems and frameworks that address streaming challenges have been developed over the past three years. In this review paper, we present the state-of-the-art in this growing vital field. <s> BIB006 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. One popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. In this paper, we propose a new dynamic classifier selection (DCS) mechanism to integrate base classifiers for effective mining from data streams. The proposed algorithm dynamically selects a single “best” classifier to classify each test instance at run time. Our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. Given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. Experimental results and comparative studies demonstrate the efficiency and efficacy of our method. Such a DCS scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence. <s> BIB007 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Data stream analysis has attracted considerable attention over the past few years. Streaming data is often evolving over time. Capturing changes could be used for detecting an event or a phenomenon in various applications. Weather conditions, economical changes, astronomical and scientific phenomena are among a wide range of applications. Due to the high volume and speed of data streams, it is computationally hard to capture these changes from raw data in real-time. In this paper, we propose a novel algorithm that we term as STREAM-DETECT to capture these changes in data stream distribution and/or domain using clustering result deviation. STREAMDETECT is followed by a process of offline classification CHANGE-CLASS. This classification is concerned with the association of the history of change characteristics with the observed event or phenomenon. Experimental results show the efficiency of the proposed framework in both detecting the changes and classification accuracy. <s> BIB008 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> As advances in networking technology help to connect the distant corners of the globe and as the Internet continues to expand its influence as a medium for communications and commerce, the threat from spammers, attackers and criminal enterprises has also grown accordingly. It is the prevalence of such threats that has made intrusion detection systems-the cyberspace's equivalent to the burglar alarm-join ranks with firewalls as one of the fundamental technologies for network security. However, today's commercially available intrusion detection systems are predominantly signature-based intrusion detection systems that are designed to detect known attacks by utilizing the signatures of those attacks. Such systems require frequent rule-base updates and signature updates, and are not capable of detecting unknown attacks. In contrast, anomaly detection systems, a subset of intrusion detection systems, model the normal system/network behavior which enables them to be extremely effective in finding and foiling both known as well as unknown or ''zero day'' attacks. While anomaly detection systems are attractive conceptually, a host of technological problems need to be overcome before they can be widely adopted. These problems include: high false alarm rate, failure to scale to gigabit speeds, etc. In this paper, we provide a comprehensive survey of anomaly detection systems and hybrid intrusion detection systems of the recent past and present. We also discuss recent technological trends in anomaly detection and identify open problems and challenges in this area. <s> BIB009 </s> A survey of multiple classifier systems as hybrid systems <s> Concept Drift <s> Handling changes over time in supervised learning (concept drift) lately has received a great deal of attention, a number of adaptive learning strategies have been developed. Most of them make an optimistic assumption that the new labels become available immediately. In real sequential classification tasks it is often unrealistic due to task specific delayed labeling or associated labeling costs. We address the problem of change detectability, given, that the new labels are not available. In this analytical study we look at the space of changes from probabilistic perspective to analyze, what changes are detectable without seeing the labels and what are not. We conduct a range of experiments with real life data with simulated and natural changes to explore this detectability issue. We propose a computationally friendly detection technique, which monitors a stream of classifier outputs. We demonstrate analytically and experimentally, what types of changes are possible to detect when the labels for the new data are not available. <s> BIB010
Before entering the discussion of practical applications we consider a very specific topic of real life relevance which is known as Concept Drift in knowledge engineering domains, or non-stationary processes in signal processing and statistics domains. Most of the conventional classifiers do not take into consideration this phenomenon. Concept Drift means that the statistical dependencies between object features and its classification may change in time, so that future data may be badly processed if we maintain the same classification, because the object category or its properties will be changing. Concept drift occurs frequently in real life BIB006 . MCS are specially well suited to deal with Concept Drift. Machine learning methods in security applications (like spam filters or IDS/IPS) BIB009 or decision support systems for marketing departments BIB002 require to take into account new training data with potentially different statistical properties BIB007 . The occurrence of Concept Drift decreases the true classification accuracy dramatically. The most popular approaches are the Streaming Ensemble Algorithm (SEA) BIB001 and the Accuracy Weighted Ensemble (AWE) BIB003 . Incoming data are collected in data chunks, which are used to train new models. The individual classifiers evaluation is done on their accuracy on the new data. The best performing classifiers are selected to constitute the MCS committee in the next time epoch. As the decision rule, the SEA uses a majority voting, whereas the AWE uses a weighted voting strategy. Kotler et al. present the Dynamic Weighted Majority (DWM) algorithm BIB004 which modifies the decision combination weights and updates the ensemble according to number of incorrect decisions made by individual classifiers. When a classifier weight is too small, then it is removed from the ensemble, a new classifier is trained and added to the ensemble in its place. A difficult problem is drift detection, which is the problem of deciding that the Concept Drift has taken place. The current research direction is to propose an additional binary classifier giving the decision to rebuild the classifiers. The drift detector can be based on changes in the probability distribution of the instances BIB008 BIB005  or classification accuracy   . Not all classification algorithms dealing with concept drift require drift detection, because they can adjust the model to incoming data BIB010 [?].