A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> [Read at a RESEARCH METHODS MEETING of the SOCIETY, April 8th, 1964, Professor D. V. LINDLEY in the Chair] SUMMARY In the analysis of data it is often assumed that observations Yl, Y2, *-, Yn are independently normally distributed with constant variance and with expectations specified by a model linear in a set of parameters 0. In this paper we make the less restrictive assumption that such a normal, homoscedastic, linear model is appropriate after some suitable transformation has been applied to the y's. Inferences about the transformation and about the parameters of the linear model are made by computing the likelihood function and the relevant posterior distribution. The contributions of normality, homoscedasticity and additivity to the transformation are separated. The relation of the present methods to earlier procedures for finding transformations is discussed. The methods are illustrated with examples. <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Applied Linear Statistical Models 5e is the long established leading authoritative text and reference on statistical modeling. The text includes brief introductory and review material, and then proceeds through regression and modeling for the first half, and through ANOVA and Experimental Design in the second half. All topics are presented in a precise and clear style supported with solved examples, numbered formulae, graphic illustrations, and "Notes" to provide depth and statistical accuracy and precision. The Fifth edition provides an increased use of computing and graphical analysis throughout, without sacrificing concepts or rigor. In general, the 5e uses larger data sets in examples and exercises, and where methods can be automated within software without loss of understanding, it is so done. <s> BIB002 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Storing and using specific instances improves the performance of several supervised learning algorithms. These include algorithms that learn decision trees, classification rules, and distributed networks. However, no investigation has analyzed algorithms that use only specific instances to solve incremental learning tasks. In this paper, we describe a framework and methodology, called instance-based learning, that generates classification predictions using only specific instances. Instance-based learning algorithms do not maintain a set of abstractions derived from specific instances. This approach extends the nearest neighbor algorithm, which has large storage requirements. We describe how storage requirements can be significantly reduced with, at most, minor sacrifices in learning rate and classification accuracy. While the storage-reducing algorithm performs well on several real-world databases, its performance degrades rapidly with the level of attribute noise in training instances. Therefore, we extended it with a significance test to distinguish noisy instances. This extended algorithm's performance degrades gracefully with increasing noise levels and compares favorably with a noise-tolerant decision tree algorithm. <s> BIB003 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Are you a data mining analyst, who spends up to 80% of your time assuring data quality, then preparing that data for developing and deploying predictive models? And do you find lots of literature on data mining theory and concepts, but when it comes to practical advice on developing good mining views find little "how to" information? And are you, like most analysts, preparing the data in SAS? ::: ::: This book is intended to fill this gap as your source of practical recipes. It introduces a framework for the process of data preparation for data mining, and presents the detailed implementation of each step in SAS. In addition, business applications of data mining modeling require you to deal with a large number of variables, typically hundreds if not thousands. Therefore, the book devotes several chapters to the methods of data transformation and variable selection. ::: ::: FEATURES ::: * A complete framework for the data preparation process, including implementation details for each step. ::: * The complete SAS implementation code, which is readily usable by professional analysts and data miners. ::: * A unique and comprehensive approach for the treatment of missing values, optimal binning, and cardinality reduction. ::: * Assumes minimal proficiency in SAS and includes a quick-start chapter on writing SAS macros. ::: * CD includes dozens of SAS macros plus the sample data and the program for the book's case study. <s> BIB004 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> The poor characterisation of contaminated soils is likely to result in high costs, restricted choice in landfill disposal sites and future environmental impact. This makes that big quantities of soils are still waiting to be recovered. Until now the tools used to detect contaminated soil have been very generic, without any criteria of prioritization. Usually, simulation studies are used to classify contaminated soils, however these systems need a large quantity of data that is difficult to obtain and manage, which means that the results obtained are subject to large uncertainty. Recently, Artificial Intelligence techniques have been used to tackle this problem. In this work we propose the use of Fuzzy Expert Systems to classify the soils. Classical decision rules have shown to be interpretable, efficient, problem independent and able to treat large scale applications, but they are also recognised as highly unstable classifiers with respect to minor perturbations in the training data. In our problem, the data is subject to uncertainty, for this reason we propose the fuzzyfication of the variables. In the study many different variables have been taken into account: Physic and Chemical characteristics of the soil and pollutants, toxicological properties, and environment and social conditions. After applying Fuzzy Expert Systems at different levels, we obtain a ranking of the soils according to their risk of contamination. The results have been contrasted with another Multicriteria Decision Making system. <s> BIB005 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Until recently, local governments in Spain were using machines with rolling cylinders for verifying taximeters. However, the condition of the tires can lead to errors in the process and the mechanical construction of the test equipment is not compatible with certain vehicles. Thus, a new measurement device needs to be designed. In our opinion, the verification of a taximeter will not be reliable unless measurements taken on an actual taxi run are used. GPS sensors are intuitively well suited for this process, because they provide the position and the speed with independence of those car devices that are under test. But there are legal problems that make difficult the use of GPS-based sensors: GPS coordinate measurements do not match exactly real coordinates and, generally speaking, we are not given absolute tolerances. We can not know whether the maximum error is always lower than, for example, 7 m. However, we might know that 50% of the measurements lie on a circle with a radius of 7 m, centered on the real position. In this paper we describe a practical application where these legal problems have been solved with soft computing based technologies. In particular, we propose to characterize the uncertainty in the GPS with fuzzy techniques, so that we can reuse certain recent algorithms, formerly intended for being used in genetic fuzzy systems, to this new context. Specifically, we propose a new method for computing an upper bound of the length of the trajectory, taking into account the vagueness of the GPS data. This bound will be computed using a modified multiobjective evolutionary algorithm, which can optimize a fuzzy valued function. The accuracy of the measurements will be improved further by combining it with restrictions based on the dynamic behavior of the vehicles. <s> BIB006 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> One of the most difficult tasks in the whole KDD process is to choose the right data mining technique, as the commercial software tools provide more and more possibilities together and the decision requires more and more expertise on the methodological point of view. Indeed, there are a lot of data mining techniques available for an environmental scientist wishing to discover some model from her/his data. This diversity can cause some troubles to the scientist who often have not a clear idea of what are the available methods, and moreover, use to have doubts about the most suitable method to be applied to solve a concrete domain problem. Within the data mining literature there is not a common terminology. A classification of the data mining methods would greatly simplify the understanding of the whole space of available methods. Furthermore, most data mining products either do not provide intelligent assistance for addressing the data mining process or tend do so in the form of rudimentary “wizard-like” interfaces that make hard assumptions about the user’s background knowledge. In this work, a classification of most common data mining methods is presented in a conceptual map which makes easier the selection process. Also an intelligent data mining assistant is presented. It is oriented to provide model/algorithm selection support, suggesting the user the most suitable data mining techniques for a given problem. <s> BIB007 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Understand the need for analyses of large, complex, information-rich data sets. Identify the goals and primary tasks of the data-mining process. Describe the roots of data-mining technology. Recognize the iterative character of a data-mining process and specify its basic steps. Explain the influence of data quality on a data-mining process. Establish the relation between data warehousing and data mining. Data mining is an iterative process within which progress is defined by discovery, through either automatic or manual methods. Data mining is most useful in an exploratory analysis scenario in which there are no predetermined notions about what will constitute an "interesting" outcome. Data mining is the search for new, valuable, and nontrivial information in large volumes of data. It is a cooperative effort of humans and computers. Best results are achieved by balancing the knowledge of human experts in describing problems and goals with the search capabilities of computers. In practice, the two primary goals of data mining tend to be prediction and description. Prediction involves using some variables or fields in the data set to predict unknown or future values of other variables of interest. Description, on the other hand, focuses on finding patterns describing the data that can be interpreted by humans. Therefore, it is possible to put data-mining activities into one of two categories: Predictive data mining, which produces the model of the system described by the given data set, or Descriptive data mining, which produces new, nontrivial information based on the available data set. <s> BIB008 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> Data Mining: Practical Machine Learning Tools and Techniques offers a thorough grounding in machine learning concepts as well as practical advice on applying machine learning tools and techniques in real-world data mining situations. This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining. Thorough updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including new material on Data Transformations, Ensemble Learning, Massive Data Sets, Multi-instance Learning, plus a new version of the popular Weka machine learning software developed by the authors. Witten, Frank, and Hall include both tried-and-true techniques of today as well as methods at the leading edge of contemporary research. *Provides a thorough grounding in machine learning concepts as well as practical advice on applying the tools and techniques to your data mining projects *Offers concrete tips and techniques for performance improvement that work by transforming the input or output in machine learning methods *Includes downloadable Weka software toolkit, a collection of machine learning algorithms for data mining tasks-in an updated, interactive interface. Algorithms in toolkit cover: data pre-processing, classification, regression, clustering, association rules, visualization <s> BIB009 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Transformations required to fit technical assumptions of data mining methods <s> The discussions in the previous chapters focus on the problem of unsupervised outlier detection in which no prior information is available about the abnormalities in the data. In such scenarios, many of the anomalies found correspond to noise or other uninteresting phenomena. It has been observed [338, 374, 531] in diverse applications such as system anomaly detection, financial fraud, and Web robot detection that interesting anomalies are often highly specific to particular types of abnormal activity in the underlying application. In such cases, an unsupervised outlier detection method might discover noise, which is not specific to that activity, and therefore may not be of interest to an analyst. In many cases, different types of abnormal instances could be present, and it may be desirable to distinguish among them. For example, in an intrusion-detection scenario, different types of intrusion anomalies are possible, and the specific type of an intrusion is important information. <s> BIB010
Some data mining methods involve technical assumptions that must be hold to guarantee the validity of the results. For example, linear regression or ANOVA require normality, homo-chedasticity, and non-colinearity. Also, inductive machine learning classifiers perform badly in front of imbalanced datasets. Thus, in front of non-normal, correlated or imbalanced datasets, transformations are apparently mandatory previous to the data mining step. The vision of the authors is to better change another data mining method well adapted to the nature of the data, rather than changing data to adapt the method  BIB007 . This is mainly because, in the context of data mining, this may lead to good models, with uncertain usefulness, made of non-understandable transformed variables. Nowadays, there are so many data mining methods, that we presume this technical transformations could be complete dispensable in the preprocessing. As an example, it is enough to change linear regression by ANN to avoid normality, linearity and homochedasticity assumptions and get better predictions for non-normal, non-linear or heterochedastic data. However, we provide here details on the most popular technical transformations, just for the cases in which they make really sense to use them. Centering: Some methods, like PCA gain properties when operating over centered data. This means to have a dataset with a center of gravity in the origin of coordinates. Each variable X k is transformed in X k = X k −x k , beingx k the arithmetic mean of X k . Centering guarantees thatx k = 0. Performing PCA with centered data enables to interpret the angles formed by the vectors projecting the variables over the factorial planes as a visualization of their linear correlation, such that 90°means uncorrelated variables, and 0°means linearly associated variables. Standardizing and normalizing: This is often used to avoid variables of different magnitudes coexisting in the same data matrix, which can bias the analysis, and also invalidate comparisons among them. Also, it is used to meet technical requirements of some data mining methods that need the variables to be properly scaled to avoid numerical ill conditions (like some distance-based methods) BIB004 . All these operations move the original variable to the interval [−1, 1]. In the classical standardization transformation named Z-score standardization in BIB008 , the variable is centered and scaled to have null mean and variance equal to 1. The standardization transformation is X k = X k −x k s k ,x k being the arithmetic mean of X k and s k the standard deviation. With this transformation X k always has mean equal to 0 and variance equal to 1. Other normalization transformations can also be used to move to [−1, 1] intervals, but they do not guarantee variance equal to 1. Normalization based on the range of the variable BIB009 , also referred by some authors as min-max normalization BIB008 BIB004 : max(X k )−min(X k ) , then X k moves to the interval [a, b] BIB009 . Decimal scaling normalization BIB008 , where X k = X k 10 r , where r is the smaller integer such that max(|X k |) < 1; this is what happens when measurement expressed in m are transformed to Km, in this case with r = 3. In BIB004 , a transformation to achieve the normal distribution of the new variable is proposed, based on first ranking observations and second substituting original values of the variable by the Logarithmic transformation: Useful to achieve homochedasticity. When the variable has values too close to 0, it is important to previously translate it horizontally to avoid numerical instability. Quadratic transformation: catching quadratic relationships with the response variable. They are useful to linearize the quadratic terms in regression models for example. The same concept extends to high order transforms Y = Xr, r > 1. In BIB001 , the Box-Cox transformation is formulated as a family of transforms depending on a parameter r in the form X k = X r k −1 r , r > 0, and X k = log(X k ), r = 0. For r = 1 it simply makes a horizontal translation of the variable. In statistical modeling they are often used to eliminate skewness and normalize asymmetric distributions. See BIB002 for details. As it can be seen, the purpose of Box-Cox transforms are strictly technical and the interpretation of X k is usually missed. The rank transformation BIB004 is based on sorting the observations of the variable and enumerating them. The transformed variable accounts for the relative position of the object in this list and becomes an ordinal variable. This is useful when the ordering among individuals is important, but the distance between the individuals in consecutive positions has no particular meaning. For example, when evaluating enterprises to get the contract to manage a WWTP in the next five years, and some evaluation is obtained from the candidates, the differences in the final score are not important, but the ranking is. Also, for distance-based data mining methods, ordinal variables (like risk of eruption in a volcano: low, moderate, high, sure) can be first recodified to ordinal encoding BIB010   BIB003 and the resulting encoded variable ranked and used in the distancebased methods (like clustering) instead of the ordinal one, for better exploiting the ordinal structure of the variable in the data mining process. As said before, sometimes data is fuzzified to better include uncertainty intrinsic from data into the data mining model BIB005 . This means to transform a crisp value in a fuzzy number and requires specific fuzzy algorithms to work with them. This process is particularly natural when data coming from certain technological devices provide uncertain punctual measurements together with some evaluation of the uncertainty itself, like variability, error bounds, occurrence regions, and so on. Radar data, GPS data or SCADA data are particularly suitable for this kind of treatment BIB006 .