A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preprocessing: i problems with the data and ii preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end. <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> Data Preparation for Data Mining addresses an issue unfortunately ignored by most authorities on data mining: data preparation. Thanks largely to its perceived difficulty, data preparation has traditionally taken a backseat to the more alluring question of how best to extract meaningful knowledge. But without adequate preparation of your data, the return on the resources invested in mining is certain to be disappointing.Dorian Pyle corrects this imbalance. A twenty-five-year veteran of what has become the data mining industry, Pyle shares his own successful data preparation methodology, offering both a conceptual overview for managers and complete technical details for IT professionals. Apply his techniques and watch your mining efforts pay off-in the form of improved performance, reduced distortion, and more valuable results.On the enclosed CD-ROM, you'll find a suite of programs as C source code and compiled into a command-line-driven toolkit. This code illustrates how the author's techniques can be applied to arrive at an automated preparation solution that works for you. Also included are demonstration versions of three commercial products that help with data preparation, along with sample data with which you can practice and experiment. * Offers in-depth coverage of an essential but largely ignored subject.* Goes far beyond theory, leading you-step by step-through the author's own data preparation techniques.* Provides practical illustrations of the author's methodology using realistic sample data sets.* Includes algorithms you can apply directly to your own project, along with instructions for understanding when automation is possible and when greater intervention is required.* Explains how to identify and correct data problems that may be present in your application.* Prepares miners, helping them head into preparation with a better understanding of data sets and their limitations. <s> BIB002 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> This article surveys the contents of the workshop Post-Processing in Machine Learning and Data Mining: Interpretation, Visualization, Integration, and Related Topics within KDD-2000: The Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Boston, MA, USA, 20-23 August 2000. The corresponding web site is on www.acm.org/sigkdd/kdd2000 First, this survey paper introduces the state of the art of the workshop topics, emphasizing that postprocessing forms a significant component in Knowledge Discovery in Databases (KDD). Next, the article brings up a report on the contents, analysis, discussion, and other aspects regarding this workshop. Afterwards, we survey all the workshop papers. They can be found at (and downloaded from) www.cas.mcmaster.ca/~bruha/kdd2000/kddrep.html The authors of this report worked as the organizers of the workshop; the programme committee was formed by additional three researches in this field. <s> BIB003 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> Are you a data mining analyst, who spends up to 80% of your time assuring data quality, then preparing that data for developing and deploying predictive models? And do you find lots of literature on data mining theory and concepts, but when it comes to practical advice on developing good mining views find little "how to" information? And are you, like most analysts, preparing the data in SAS? ::: ::: This book is intended to fill this gap as your source of practical recipes. It introduces a framework for the process of data preparation for data mining, and presents the detailed implementation of each step in SAS. In addition, business applications of data mining modeling require you to deal with a large number of variables, typically hundreds if not thousands. Therefore, the book devotes several chapters to the methods of data transformation and variable selection. ::: ::: FEATURES ::: * A complete framework for the data preparation process, including implementation details for each step. ::: * The complete SAS implementation code, which is readily usable by professional analysts and data miners. ::: * A unique and comprehensive approach for the treatment of missing values, optimal binning, and cardinality reduction. ::: * Assumes minimal proficiency in SAS and includes a quick-start chapter on writing SAS macros. ::: * CD includes dozens of SAS macros plus the sample data and the program for the book's case study. <s> BIB004 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> This comprehensive textbook on data mining details the unique steps of the knowledge discovery process that prescribes the sequence in which data mining projects should be performed, from problem and data understanding through datapreprocessing to deployment of the results. This knowledge discovery approach is what distinguishes Data Mining from other texts in this area. The book provides a suite of exercises and includes links to instructional presentations. Furthermore, it containsappendices of relevant mathematical material. <s> BIB005 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> Abstract Over recent years a huge library of data mining algorithms has been developed to tackle a variety of problems in fields such as medical imaging and network traffic analysis. Many of these techniques are far more flexible than more classical modelling approaches and could be usefully applied to data-rich environmental problems. Certain techniques such as Artificial Neural Networks, Clustering, Case-Based Reasoning and more recently Bayesian Decision Networks have found application in environmental modelling while other methods, for example classification and association rule extraction, have not yet been taken up on any wide scale. We propose that these and other data mining techniques could be usefully applied to difficult problems in the field. The chapter is a general introduction to Data Mining techniques for Environmental Scientists who may be interested in using them in their applications. So the presentation focuses on the contributions of data mining techniques to environmental applications and on general guidelines of good practice in real world domains. The purpose of this chapter is not to present technical details on specific data mining techniques, but rather to provide general guidance to non-expert users to help them decide which technique is appropriate for solving their problem. References to the wider literature are provided. <s> BIB006 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> The quality of discovered knowledge is highly depending on data quality. Unfortunately real data use to contain noise, uncertainty, errors, redundancies or even irrelevant information. The more complex is the reality to be analyzed, the higher the risk of getting low quality data. Knowledge Discovery from Databases (KDD) offers a global framework to prepare data in the right form to perform correct analyses. On the other hand, the quality of decisions taken upon KDD results, depend not only on the quality of the results themselves, but on the capacity of the system to communicate those results in an understandable form. Environmental systems are particularly complex and environmental users particularly require clarity in their results. In this paper some details about how this can be achieved are provided. The role of the pre and post processing in the whole process of Knowledge Discovery in environmental systems is discussed. <s> BIB007 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> Data Mining: Practical Machine Learning Tools and Techniques offers a thorough grounding in machine learning concepts as well as practical advice on applying machine learning tools and techniques in real-world data mining situations. This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining. Thorough updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including new material on Data Transformations, Ensemble Learning, Massive Data Sets, Multi-instance Learning, plus a new version of the popular Weka machine learning software developed by the authors. Witten, Frank, and Hall include both tried-and-true techniques of today as well as methods at the leading edge of contemporary research. *Provides a thorough grounding in machine learning concepts as well as practical advice on applying the tools and techniques to your data mining projects *Offers concrete tips and techniques for performance improvement that work by transforming the input or output in machine learning methods *Includes downloadable Weka software toolkit, a collection of machine learning algorithms for data mining tasks-in an updated, interactive interface. Algorithms in toolkit cover: data pre-processing, classification, regression, clustering, association rules, visualization <s> BIB008 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> The importance of post-processing the results of clustering when using data mining to ::: support subsequent decision-making is discussed. Both the formal embedded binary logistic ::: regression (EBLR) and the visual profile’s assessment grid (PAG) methods are presented ::: as bridging tools for the real use of clustering results. EBLR is a sequence of logistic ::: regressions that helps to predict the class of a new object; while PAG is a graphical tool that ::: visualises the results of an EBLR. PAG interactively determines the most suitable class for a ::: new object and enables subsequent follow-ups. PAG makes the underlying mathematical ::: model (EBLR) more understandable, improves usability and contributes to bridging the gap ::: between modelling and decision-support. When applied to medical problems, these tools ::: can perform as diagnostic-support tools, provided that the predefined set of profiles refer ::: to different stages of a certain disease or different types of patients with a same medical ::: problem, etc. Being a graphical tool, PAG enables doctors to quickly and friendly determine ::: the profile of a patient in the everyday activity, without necessarily understanding the ::: statistical models involved in the process, which used to be a serious limitation for wider ::: application of these methods in clinical praxis. In this work, an application is presented ::: with 4 functional disability profiles. <s> BIB009 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> In this paper, we propose a new visualization approach based on a Sensitivity Analysis (SA) to extract human understandable knowledge from supervised learning black box data mining models, such as Neural Networks (NNs), Support Vector Machines (SVMs) and ensembles, including Random Forests (RFs). Five SA methods (three of which are purely new) and four measures of input importance (one novel) are presented. Also, the SA approach is adapted to handle discrete variables and to aggregate multiple sensitivity responses. Moreover, several visualizations for the SA results are introduced, such as input pair importance color matrix and variable effect characteristic surface. A wide range of experiments was performed in order to test the SA methods and measures by fitting four well-known models (NN, SVM, RF and decision trees) to synthetic datasets (five regression and five classification tasks). In addition, the visualization capabilities of the SA are demonstrated using four real-world datasets (e.g., bank direct marketing and white wine quality). <s> BIB010 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Introduction <s> In last years, mining financial data has taken remarkable importance to complement classical techniques. Knowledge Discovery in Databases provides a framework to support analysis and decision-making regarding complex phenomena. Here, clustering is used to mine financial patterns from Venezuelan Stock Exchange assets (Bolsa de Valores de Caracas), and two major indexes related to that market: Dow Jones (USA) and BOVESPA (Brazil). Also, from a practical point of view, understanding clusters is crucial to support further decision-making. Only few works addressed bridging the existing gap between the raw data mining (DM) results and effective decision-making. Traffic lights panel (TLP) is proposed as a post-processing tool for this purpose. Comparison with other popular DM techniques in financial data, like association rules mining, is discussed. The information learned with the TLP improves quality of predictive modelling when the knowledge discovered in the TLP is used over a multiplicative model including interactions. <s> BIB011
Environmental systems (ESs) typically contain many interrelated components and processes, which may be biological, physical, geological, climatic, chemical, or social. Whenever we attempt to analyze ESs and related problems, we are immediately confronted with complexity stemming from various sources. Thus, there is a great need for data analysis, modeling of ESs and development of decision support systems in order to improve the understanding of ESs behavior and the management of these complex systems (specially under abnormal situations). As stated in BIB006 , the special features of environmental processes demand a careful approach to improve the analysis to be better known, modeled and consequently better managed or controlled. * Corresponding author. E-mail: karina.gibert@upc.edu. Knowledge Discovery of Data (KDD) BIB005 BIB008 appeared in 1989  referring to high level applications including particular methods of Data Mining (DM, see Fig. 1 ), oriented to extract useful and understandable knowledge from complex data. Thus, KDD is a specifically appealing umbrella to analyze environmental data. Providing highly useful information from data bases is per se very important, although KDD commonly is a preparatory activity for an environmental software system development. Also, the KDD approach facilitates the integration of different knowledge sources and fields of expertise and the involvement of end-user (domain expert) criteria and stakeholders' points of view in algorithm design and result interpretation, which bridges the gap between data and real effective decision-making levels. Thus, in the last years an increasing interest to apply DM to environmental data has been observed. Fayyad Fig. 1 . Outline of the knowledge discovery from data process as originally defined by Fayyad  . terior analysis as much important as the application of DM techniques themselves  : Most previous work on KDD has focused on [. . . ] DM step. However, the other steps are of considerable importance for the successful application of KDD in practice. In fact, both prior and posterior analyses require great effort when dealing with real applications  BIB007 BIB009 . Prior analysis is critical, mainly owing to two reasons: • Real data sets tend to be imperfect, contain errors, outliers, missing data, extra noise. Tools either for detecting or correcting them are required. • Application of a certain data mining technique may require specific conditions for the data set (only binary variables, centered data, normality, only qualitative variables, etc.). In this case, tools for verifying that those conditions hold, or eventually transform data in the appropriate way to meet these conditions, are required. In environmental data, measurement errors (from automatic or manual monitoring), uncertainty, imprecision, multi-scalarity, non-linearities, non-stacionarity, non-normality, heterogeneity, etc., are frequent. Also, redundant variables, irrelevant, or even contradictory, are found. Systematic data pre-processing including objective exploration, visualization and data transformation is particularly critical for: • Better understanding the data set; • Detecting imperfections in the data and managing them in the proper way to guarantee a correct analysis; • Correctly preparing data for the selected DM technique/s, if the required assumptions do not hold; • The correctness of the DM itself, which critically depends on the quality of the data and wrong or poor pre-processing may lead to incorrect results. Also, particular efforts in post-processing the results provided by a DM technique are important in this context, in order to make these results directly understandable by an environmental scientist, who has to make real decisions upon them, ensuring a real impact to the environmental system behavior BIB003 BIB010  BIB011 BIB009 . In fact, it can be said that the quality of the decisions will depend, not only on the quality of the data mining results themselves, but also on the capacity of the system to communicate the relevant results to the decision-maker as understandably as possible, and, in a first level, on the data quality itself (Fig. 2) BIB002 . Indeed, data cleaning  , also known as data preparation BIB002 or pre-processing BIB001 , are often time consuming and difficult; mainly because the approaches taken should be tailored to each specific application, and human interaction is required. In fact, once pre-processing is finished and data is ready for the analysis, the application of DM algorithms becomes quick (requiring some parameter settings in the appropriate software), and can be often automated. In fact, a small proportion of the time devoted to the whole KDD process is spent in the DM step. In real applications, the time devoted to pre-processing is rarely below 70% of the time-life for the whole KDD process  BIB004 . Anyway, a serious pre-processing is essential for obtaining good quality results, and useful new knowl- edge from it. Data miners should become conscious of the importance of performing very careful and rigorous pre-processing, and allocate sufficient time to this activity accordingly. Most part of the research in Data Mining has been basically focused on providing better data mining algorithms, but few works specifically focus on both pre and post-processing, despite both are recognized as critical steps for successful KDD processes by the scientific community. In this work we focus on the very first step of the process, related to pre-processing in environmental applications. A clear methodology for tackling pre-processing within the KDD process has not been well-established yet. This paper does not intend to be exhaustive, but to provide environmental scientists with tools for addressing the most common problems arisen in pre-processing as well as to provide a specific methodological proposal. Authors expect this paper can contribute to make pre-processing fully available to both data miners and environmental scientists and to help them to better pre-process their data, getting better KDD results, and as a consequence, better knowledge from their environmental systems. This paper tries to provide a reference material of what to do in practice, although work is still required to provide systematic decision criteria in pre-processing.