A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Determining the data matrix columns <s> The quality of discovered knowledge is highly depending on data quality. Unfortunately real data use to contain noise, uncertainty, errors, redundancies or even irrelevant information. The more complex is the reality to be analyzed, the higher the risk of getting low quality data. Knowledge Discovery from Databases (KDD) offers a global framework to prepare data in the right form to perform correct analyses. On the other hand, the quality of decisions taken upon KDD results, depend not only on the quality of the results themselves, but on the capacity of the system to communicate those results in an understandable form. Environmental systems are particularly complex and environmental users particularly require clarity in their results. In this paper some details about how this can be achieved are provided. The role of the pre and post processing in the whole process of Knowledge Discovery in environmental systems is discussed. <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Determining the data matrix columns <s> The Millennium Development Goals set the target to “halve by 2015 the proportion of people without sustainable access to safe drinking water and basic sanitation”. The current international definition of indicators of access is insufficient for measuring this target. Furthermore, the lack of an internationally agreed definition and measurement methodology is causing confusion and uncertainty regarding the figures that are disseminated worldwide. Moreover, the current context, in which almost 70% of funds for the sector are channelled through national governments, emphasises the importance of a monitoring system for national water sectors in developing countries. From this, an improvement in investment efficiency is expected. The water point mapping methodology, promoted and widely developed by the international organization WaterAid, is presented as an alternative way of defining water access indicators. The present paper describes its potential for defining new indicators and making improvements. Our research is based on the results of a pilot study carried out by Ingenieria Sin Fronteras (Engineering without Borders, Spain) in the Same district, Tanzania. The paper concludes with a discussion of the challenges posed by the implementation of this methodology and proposes further steps for improvement. <s> BIB002
The second issue is to determine the subset of columns of the data matrix to be considered for the analysis. This is related to the goals of the analysis and the kind of questions that have to be answered with the analysis, or the decisions to be made afterwards. Datasets may contain irrelevant variables regarding to these goals, questions or decisions as well as redundant variables BIB001 . As previously stated, the quality of discovered knowledge usually depends on the quality of the data used. Also, the success of some learning schemes, in their attempts to construct models of data, often relies on the identification of a small set of highly predictive variables. The inclusion of irrelevant, redundant and noisy variables in the model can result in poor predictive performance and increased computation. This is the reason why a previous work for determining the variables to be kept is required. At this stage of the analysis an expert-guided variable selection process is to be conducted. This means that, according to the goals of the analysis, the existing knowledge on the targeted Environmental System and the meaning of the available variables, a first selection is performed. This first step is interactive with the experts. For example, given a database with information about water infrastructures in a set of villages BIB002 , with an interest in improving drinking water access to households, all variables describing the status of the sewer network infrastructures can be dismissed. In case of doubt, it is preferable to be maximalist and keep a variable that will be disregarded later along the analysis, than to find during the data mining step that some variables initially dismissed become required. The pre-processing can imply several transformations on the data matrix (like selection of some rows) that may complicate the addition of other variables in an intermediate point of the modeling process. Re-processing a certain data mining model without some variable has a much lower computational cost than re-pre-processing the whole database. 