A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> K. Gibert et al. / A survey on pre-processing techniques: Relevant issues in the context of environmental data mining 649 <s> This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preprocessing: i problems with the data and ii preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end. <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> K. Gibert et al. / A survey on pre-processing techniques: Relevant issues in the context of environmental data mining 649 <s> Before applying learning algorithms to datasets, practitioners often globally discretize any numeric attributes. If the algorithm cannot handle numeric attributes directly, prior discretization is essential. Even if it can, prior discretization often accelerates induction, and may produce simpler and more accurate classi ers. As it is generally done, global discretization denies the learning algorithm any chance of taking advantage of the ordering information implicit in numeric attributes. However, a simple transformation of discretized data preserves this information in a form that learners can use. We show that, compared to using the discretized data directly, this transformation signi cantly increases the accuracy of decision trees built by C4.5, decision lists built by PART, and decision tables built using the wrapper method, on several benchmark datasets. Moreover, it can signi cantly reduce the size of the resulting classi ers. This simple technique makes global discretization an even more useful tool for data preprocessing. <s> BIB002 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> K. Gibert et al. / A survey on pre-processing techniques: Relevant issues in the context of environmental data mining 649 <s> Are you a data mining analyst, who spends up to 80% of your time assuring data quality, then preparing that data for developing and deploying predictive models? And do you find lots of literature on data mining theory and concepts, but when it comes to practical advice on developing good mining views find little "how to" information? And are you, like most analysts, preparing the data in SAS? ::: ::: This book is intended to fill this gap as your source of practical recipes. It introduces a framework for the process of data preparation for data mining, and presents the detailed implementation of each step in SAS. In addition, business applications of data mining modeling require you to deal with a large number of variables, typically hundreds if not thousands. Therefore, the book devotes several chapters to the methods of data transformation and variable selection. ::: ::: FEATURES ::: * A complete framework for the data preparation process, including implementation details for each step. ::: * The complete SAS implementation code, which is readily usable by professional analysts and data miners. ::: * A unique and comprehensive approach for the treatment of missing values, optimal binning, and cardinality reduction. ::: * Assumes minimal proficiency in SAS and includes a quick-start chapter on writing SAS macros. ::: * CD includes dozens of SAS macros plus the sample data and the program for the book's case study. <s> BIB003
Most of the software tools provide automatic methods to discretize numerical variables (by equal width bins, or equal frequency groups, given the number of desired groups) BIB003 . However, despite this transform can increase model performance, the meaning of the resulting beans often becomes difficult to understand and the data mining loses interpretability. In fact, in real applications it is quite common to globally discretize any numerical attribute before applying learning algorithms to datasets, since a number of them cannot handle numerical variables directly. Some data mining techniques increase performance with prior discretization, like decision trees BIB002 . However, the reader must know that discretization is changing the inner structure of the problem and the results provided by the discretized data can be significantly different from those provided by the original data, even if the intervals are well justified and consensual. This is the case of  , where concentration of hormones in blood are discretized to Low, Normal and High to identify profiles of thyroids dysfunctions and it is shown that the models obtained with the original quantitative concentrations are better, even if the discretization is based on WHO reference values. Noise is often a critical issue, and especially with environmental data, some bias may exist that can be removed with a filter BIB001 . Transformations should always be justified and documented, and the biases that may be introduced noted. It is strongly recommended to keep the interpretability of transformed variables.