An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Abstract A neon circuit which consists of nonvolatile metal-ferroelectric-semiconductor field effect transistors (MFSFETs) and a uni-junction transistor (UJT) has been proposed. In the proposed circuit MFSFETs act as analog memories to store the synaptic weights which can be changed by the adaptive-learning process during the operations. In this paper, we first simulate the operation of the ferroelectric neuron circuit using a circuit simulator, SPICE. It is shown that the output frequency of the proposed neuron circuit can be changed after it processes a certain number of input pulses. Then, we report the fabrication of UJTs and UJT pulse oscillation circuits using silicon-on-insulator (SOI) substrates. It is found that the output frequency increases with decreasing the charging time of the capacitor in the circuit and that the operation at higher frequencies is possible for integrated UJT oscillation circuits. Finally, we demonstrate the memory and learning properties of n-channel ferroelectric-gate FETs using (Pb,La)(Zr,Ti)O3 (PLZT) films. It is shown that the drain current of the PLZT/SrTiO3/Si FETs can be controlled by a “write” pulse before the measurements. <s> BIB001 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Ensemble methods have recently garnered a great deal of attention in the machine learning community. Techniques such as Boosting and Bagging have proven to be highly effective but require repeated resampling of the training data, making them inappropriate in a data mining context. The methods presented in this paper take advantage of plentiful data, building separate classifiers on sequential chunks of training points. These classifiers are combined into a fixed-size ensemble using a heuristic replacement strategy. The result is a fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift. <s> BIB002 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Most of the work in machine learning assume that examples are generated at random according to some stationary probability distribution. In this work we study the problem of learning when the distribution that generate the examples changes over time. We present a method for detection of changes in the probability distribution of examples. The idea behind the drift detection method is to control the online error-rate of the algorithm. The training examples are presented in sequence. When a new training example is available, it is classified using the actual model. Statistical theory guarantees that while the distribution is stationary, the error will decrease. When the distribution changes, the error will increase. The method controls the trace of the online error of the algorithm. For the actual context we define a warning level, and a drift level. A new context is declared, if in a sequence of examples, the error increases reaching the warning level at example k w , and the drift level at example k d . This is an indication of a change in the distribution of the examples. The algorithm learns a new model using only the examples since k w . The method was tested with a set of eight artificial datasets and a real world dataset. We used three learning algorithms: a perceptron, a neural network and a decision tree. The experimental results show a good performance detecting drift and with learning the new concept. We also observe that the method is independent of the learning algorithm. <s> BIB003 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> One major problem of existing methods to mine data streams is that it makes ad hoc choices to combine most recent data with some amount of old data to search the new hypothesis. The assumption is that the additional old data always helps produce a more accurate hypothesis than using the most recent data only. We first criticize this notion and point out that using old data blindly is not better than "gambling"; in other words, it helps increase the accuracy only if we are "lucky." We discuss and analyze the situations where old data will help and what kind of old data will help. The practical problem on choosing the right example from old data is due to the formidable cost to compare different possibilities and models. This problem will go away if we have an algorithm that is extremely efficient to compare all sensible choices with little extra cost. Based on this observation, we propose a simple, efficient and accurate cross-validation decision tree ensemble method. <s> BIB004 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> An emerging problem in Data Streams is the detection of concept drift. This problem is aggravated when the drift is gradual over time. In this work we deflne a method for detecting concept drift, even in the case of slow gradual change. It is based on the estimated distribution of the distances between classiflcation errors. The proposed method can be used with any learning algorithm in two ways: using it as a wrapper of a batch learning algorithm or implementing it inside an incremental and online algorithm. The experimentation results compare our method (EDDM) with a similar one (DDM). Latter uses the error-rate instead of distance-error-rate. <s> BIB005 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> This paper proposes a general framework for classifying data streams by exploiting incremental clustering in order to dynamically build and update an ensemble of incremental classifiers. To achieve this, a transformation function that maps batches of examples into a new conceptual feature space is proposed. The clustering algorithm is then applied in order to group different concepts and identify recurring contexts. The ensemble is produced by maintaining an classifier for every concept discovered in the stream The full version of this paper as well as the datasets used for evaluation can be found at: http://mlkd.csd.auth.gr/concept_drift.html <s> BIB006 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Concept drift constitutes a challenging problem for the machine learning and data mining community that frequently appears in real world stream classification problems. It is usually defined as the unforeseeable concept change of the target variable in a prediction task. In this paper, we focus on the problem of recurring contexts, a special sub-type of concept drift, that has not yet met the proper attention from the research community. In the case of recurring contexts, concepts may re-appear in future and thus older classification models might be beneficial for future classifications. We propose a general framework for classifying data streams by exploiting stream clustering in order to dynamically build and update an ensemble of incremental classifiers. To achieve this, a transformation function that maps batches of examples into a new conceptual representation model is proposed. The clustering algorithm is then applied in order to group batches of examples into concepts and identify recurring contexts. The ensemble is produced by creating and maintaining an incremental classifier for every concept discovered in the data stream. An experimental study is performed using (a) two new real-world concept drifting datasets from the email domain, (b) an instantiation of the proposed framework and (c) five methods for dealing with drifting concepts. Results indicate the effectiveness of the proposed representation and the suitability of the concept-specific classifiers for problems with recurring contexts. <s> BIB007 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Concept drift refers to a non stationary learning problem over time. The training and the application data often mismatch in real life problems. In this report we present a context of concept drift problem 1. We focus on the issues relevant to adaptive training set formation. We present the framework and terminology, and formulate a global picture of concept drift learners design. We start with formalizing the framework for the concept drifting data in Section 1. In Section 2 we discuss the adaptivity mechanisms of the concept drift learners. In Section 3 we overview the principle mechanisms of concept drift learners. In this chapter we give a general picture of the available algorithms and categorize them based on their properties. Section 5 discusses the related research fields and Section 5 groups and presents major concept drift applications. This report is intended to give a bird's view of concept drift research field, provide a context of the research and position it within broad spectrum of research fields and applications. <s> BIB008 </s> An Overview on Concept Drift Learning <s> 1) SYNTHETIC DATASETS <s> Concept drift primarily refers to an online supervised learning scenario when the relation between the input data and the target variable changes over time. Assuming a general knowledge of supervised learning in this article, we characterize adaptive learning processes; categorize existing strategies for handling concept drift; overview the most representative, distinct, and popular techniques and algorithms; discuss evaluation methodology of adaptive algorithms; and present a set of illustrative applications. The survey covers the different facets of concept drift in an integrated way to reflect on the existing scattered state of the art. Thus, it aims at providing a comprehensive introduction to the concept drift adaptation for researchers, industry analysts, and practitioners. <s> BIB009
Below, some synthetic datasets used in literature: • SINE1: abrupt concept drift, noise-free instances. Two relevant attributes, each one with a uniformly distributed value in [0, 1]. In the first concept, it classifies as positive if a value stands below the curve y = sin(x); is classified as negative otherwise. After drift, the classification is reversed BIB003 . • SINE2: has two relevant attributes like SINE1. Classification function is y < 0.5 + 0.3 × sin(3π x). After drift, the concept is inverted BIB003 . • SINIRREL1: has classification function as SINE1, the instances, however, present two irrelevant attributes BIB003 . • SINIRREL2: has classification function as SINE2, but like SINIRREL1, instances present two irrelevant attributes BIB003 . • CIRCLES: presents gradual drift and instances without noise. It has four classification function determined by four circles (four concepts). Instances are classified according to its location: if it is inside the circle defined by circular function, it is classified as positive; otherwise is negative. The gradual change occurs by modifying the center and radius size of the circle BIB003 . • GAUSS: has abrupt drift and instances with noise. Domain R × R with two relevant attributes. Positive instances are located as a normal distribution with center [0, 0] and standard deviation as of 1. Negative instances have center BIB008 0] and standard deviation as of 4. After drift, the classification is reversed BIB003 . • SINE1G: presents very slow gradual drift and instances without noise. Similar to Sine1, but the gradual drift is reached by selecting instances from the past and the current concept (transition time among concepts). To select an instance from the past concept and the new one has, respectively, gradually lower probability and gradually higher probability as time passes BIB005 . • STAGGER: has abrupt drift and instances without noise. Instances have three symbolic attributes -size (small, medium, large), color (red, green), and shape (circular, non-circular). In the first concept, instances are positive if size = small ∧ color = red. In second concept the instances are defined by color = green ∨ shape = circular. In third concept the instances are defined by size = medium ∨ size = large BIB003 . • MIXED: has abrupt drift and instances without noise. Instances have four attributes: two boolean v, w and two numeric between [0, 1]. If an instance has the following two of three conditions satisfied, it is classified as positive: v, w, y < 0.5 + 0.3 × sin(3π x). After drift, classification is reversed BIB003 . • Rotating Hyperplane Dataset: designed by BIB004 . The (k, t) pairs details of each concept and all files are available in the Internet 1  . • Usenet1 and Usenet2: used in BIB006 , are available in the Internet. BIB008 They collected reports from several newsgroups (e.g., medicine, space, baseball) of a user. The distinction among datasets is the drift dimension: the Usenet1 dataset has a sharper topic drift  . • Usenet: text dataset inspired by Katakis et al. BIB007 , available in the Internet. BIB009 Usenet simulates a news filtering from 20 Newsgroups with change of interest of a user (concept drift). There are six topics, the user is interested 1 http://www.win.tue.nl/mpechen/data/ DriftSets/ 2 http://mlkd.csd.auth.gr/concept_drift.html 3 http://www.liaad.up.pt/kdus/products/datasets-for-concept-drift in two, but is subscribed in four. It also simulates recurring concepts repeating topics of interest (three concepts in training data and three recurring concepts in testing data). The dataset has 5,931 instances and 659 attributes, which are binary values (presence or absence of the respective word). • SEA Concepts Dataset: proposed by Street and Kim BIB002 , this dataset has 60,000 instances, 3 attributes, and 3 classes; with 10% of noise. The numeric attributes are between 0 and 10 with two relevant attributes. Instances are divided into groups of 15,000 into four concepts. Each concept has different thresholds values (8, 9, 7, and 9.5) , and the concept function to determine 0 to a class instance is relevant_feature1 + relevant_feature2 > Threshold. Dataset is available in the Internet, BIB001 and is quite used by concept drift handling algorithms  .