A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Error detection and treatment <s> This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preprocessing: i problems with the data and ii preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end. <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Error detection and treatment <s> IT IS CLEAR THAT NOWADAYS ANALYSIS OF COMPLEX SYSTEMS IS AN IMPORTANT HANDICAP FOR EITHER STATISTICS, ARTIFICIAL INTELIGENCE, INFORMATION SYSTEMS, DATA VISUALIZATION""| DESCRIBING THE STRUCTURE OR OBTAINING KNOWLWDGE FROM COMPLEX SYSTEMS IS KNOWN AS A DIFFICULT TASK. IT IS INEGABLE THAT THE COMBINATION OF DATA ANALYSIS TECHNIQUES (CLUSTERING AMONG THEM), INDUCTIVE LAERNING (KNOWLEDGE - BASED SYSTEMS), MANAGEMENT OF DATA BASES AND MULTIMENSIONAL GRAPHICAL REPRESENTATION MUST PRODUCE BENEFITS ON THIS LINE. FACTING THE AUTOMATED KNOWLEDGE DISCOVERY OF ILL - STRUCTURED DOMAINS RAISES SOME PROBLEMS EITHER FROM A MACHINE LEARNING OR CLUSTERING POINT OF VIEW. CLUSTERING BASED ON RULES (CBR) IS A METHODOLOGY DEVELOPED IN (9) WITH THE AIM OF FIDING THE STRUCTURE OF ILL - STRUCTURED DOMAINS. IN OUR PROPOSAL, A COMBINATION OF CLUSTERING AND INDUCTIVE LEARNING IS FOCUSSED TO THE PROBLEM OF FINDING AND INTERPRETING SPECIAL PATTERNS (OR CONCEPTS) FROM LARGE DATA BASES, IN ORDER TO EXTRACT USEFUL KNOWLEDGE TO REPRESENT REAL - WORD DOMAINS, GIVING BETTER PERFOMANCE THAN TRADITIONAL CLUSTERING ALGORITHMS OR KNOWLWDGE BASED SYSTEMS APPROACH. THE SCOPE OF THIS PAPER IS TO PRESENT THE METHODOLOGY ITSELF AS WELL AS TO SHOW HOW CBR HAS SEVERAL CONNECTIN POINTS WITH KNOWLEDGE DISCOVERY OF DATA. SOME APPLICATIONS ARE USED TO ILLUSTRATE THIS IDEAS. <s> BIB002
Corrupt data can come from sensor failure, data transmission, improper data entry, etc. Often, it produces values out of range BIB001 . Corrupt data can also be identified by evaluating consistency rules over the data, representing wellknown relationships among the variables. As an example, in Fig. 4 , the mistake in BOD = 1279 can be identified by applying specific domain knowledge: In fact, BOD and COD are correlated and it is not possible that one is high and the other one is low simultaneously. To see which of both is wrong, other variables help, like suspended solids, also positively correlated with them. To identify corrupt data, specific domain knowledge is required together with intensive interaction with the expert. When an error is detected, they are basically two ways of treating it: • Correction: This means to come back to the original data source and try to retrieve the correct data. This is the preferable option, when available. • Missing production: When correction is not available, the data must be substituted by a missing value and conveniently imputed. A special task in this topic is terminology normalization for qualitative variables. More and more, qualitative variables play a bigger role in data mining. It often happens that a same modality is expressed with different strings, which are not properly recognized by the software as equal, thus significantly altering the data mining results. For example, data coming from different countries can have modalities expressed in different languages (a qualitative variable indicating risk of fire can have Alt, Alto, High, Elevé, all of them meaning high in different languages; they will be treated as different in the analysis if they are not properly preprocessed). In a more specific situation, as common Thesaurus are not extensively used yet to fill-in qualitative variables, slight differences in the words spelling might have the same consequences (accents, blank characters, capital letters, special characters like dashes, slashes or points are the biggest enemies of qualitative variables' treatment). As an example, in the dataset containing the measurements from Hipparcos satellite, one of the variables is the Spectrum of the measured star BIB002 . The file contains measurements over 87475 stars, and the descriptive analysis says that they are 3655 different Spectrum values. Some of the stars show a Spectrum type "M2 :", while other show "M2 : " or even "M2:", which in fact are the same, but codified in different strings, and the software do not identify as equal. Also "M3 J " and "M3 J ". The difference is subtle, only some blank spaces distinguishing them, but the effect over data mining results is important. Also "A0p (Si)", or "A0p Si" and also "A5 IV-V" or "A5IV/V". After standardizing the labels of all modalities of the variable, the number of modalities reduces to some hundreds. So, very careful editing is required to normalize terminology before starting the analysis. Any decision made for data error handling must be transparently reported.