A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Creating new variables <s> This paper is an introduction of Knowledge Discovery in Serial Measurement (KDSM) methodology for analyzing repeated and very short serial measures with a blocking factor in ill-structured domains (ISD). ::: ::: KDSM arises from the results obtained in a real application of psychiatry (presented in the previous issue of CCIA [11]). In this application domain, common statistical analysis (time series analysis, multivariate data analysis...) and artificial intelligence techniques (knowledge based methods, inductive learning), employed independently, are often inadequate due to the intrinsic characteristics of ISD. ::: ::: KDSM is based on both the combination of statistical methods and artificial intelligence techniques, including the use of clustering based on rules (introduced by Gibert in 1994). <s> BIB001 </s> A survey on pre-processing techniques: Relevant issues in the context of environmental data mining <s> Creating new variables <s> This work presents advances in the design of a hybrid methodology that combines artificial intelligence and statistical tools to induce a model of explicit knowledge in relation to the dynamics of a wastewater treatment plant. The methodology contributes to problem solving under the paradigm of knowledge discovery from data in which the pre-process, the automatic interpretation of results and the explicit production of knowledge play a role as important as the analysis itself. The data mining step is performed using clustering based on rules by states, which integrates the knowledge discovered separately at each step of the process into a single model of global operation of the phenomenon. This provides a more accurate model for the dynamics of the system than one obtained by analyzing the whole dataset with all the steps taken together. <s> BIB002
Finally, in KDD it is quite usual to build new variables by combining some others included in the dataset. Here, expert knowhow and domain knowledge is usually the guide. Often new variables are nonlinear combinations of the measured variables, thus providing new elements for the modeling step that can significantly increase model performances. When the purpose of these transformations is to make explicit some of the decisional variables that the user really has in mind for decision-making, this is a good practice, bringing significantly added value to the dataset, and opening the way to better data mining models. There are different scenarios: Creation of aggregates. Like totals. As an example, consider the concentrations of several pollutants in atmosphere, like CO 2 , nitrogen, arsenic, selenium, antimony in atmosphere, it makes sense to build a new variable with total concentration of heavy metals, by adding concentrations of arsenic, selenium and antimony. This might increase interpretability of the data mining models obtained. However, care is required, as this decreases the granularity of information, by collapsing different profiles of air quality; whereas decisions associated with exceeding of arsenic should be different of exceeding of cadmium, using a global variable for all heavy metals together is masking the possibility to understand what is happening in detail. The same happens if all pollutants found in the outflow of a WWTP are added together in a variable concentration of pollutants BIB002 . In that case, a high level of this variable do not permit to understand if the plant has abnormal operations in the settler (producing too high suspended solids concentrations) or in the bioreactor (too high organic matter concentrations). Another operation that merits attention in pre-processing is what we can group under the term knowhowbased feature extraction, where new variables will appear on the database by combining several original variables in specific ways indicated by the experts. As said before, explicit creation of new variables that approach the decisional criteria used by the experts might provide a powerful set of variables that improve performance of data mining models. This practice often increases also usability, as providing models in terms of the reasoning variables used by experts, and ap- 4000 μS/cm, D = 0 otherwise)  and, as it can be seen, ISQA is a non-linear relationship among these parameters. Introducing it as an additional variable in data mining modeling can improve the performance and interpretability of the model. Another example is to build the atmospheric clearness index as the ratio between two other variables that can be directly measured: the solar radiation and ground level to extra-terrestrial solar radiation. Creation of binary indicators: evaluating to TRUE or FALSE according to a certain Boolean condition expressed over some subset of observed variables. For example, a variable Abnormal operation of the bioreactor in a WWTP, that will evaluate to TRUE if the effluent of the plant has concentrations of organic matter over the thresholds permitted by the law or there was bulking or foaming in the bioreactor. Or lethal radiation that will evaluate to TRUE if the radiation is lethal in some weeks or lethal in some hours or lethal with immediate effect. Another type of operation is related to creation of new variables by split or concatenation of the original ones: • Splitting a variable: It creates a vector of variables X k by decomposing a variable Y in parts. As an example, the EWC  is an international European code for encoding waste, which has a format that reflects the taxonomy in behind. Thus, the se-mantics of the several groups of digits of the code correspond to the categories of the waste from a more general classification to a more specific. As an example, code 050105 has the following meaning: first two digits 05 indicates a group of sources generating similar kinds of waste, in this case Wastes from petroleum refining, natural gas purification and pyrolytic treatment of coal. Third and fourth digits indicate a more specific sector inside the main source. In the example 01 means wastes from petroleum refining. Finally, the last two digits give the specific waste. Special attention is required for multivalued variables, introduced in Section 3.1. From the informational point of view, the s binary variables, representing the modalities of the multivalued variable, are providing information of a single aspect of the object described in each data row. An example is the variable technologies available in a city to provide access to water expressed by means of s = 3 binary variables in  . However, if the multivalued variable is introduced in the data mining analysis as a set of s independent binary variables, this artificially increases the dimensionality of the data matrix by dedicating s dimensions to a single variable. This often biases the analysis. In the example, it is like if the water access technologies were more important in the description of the cities than other variables, like population density or type of climate, both explicable in a single column each. To avoid this excess of weight of the multivalued variable, some possibilities are available: one is to build the concatenation of all binary components, as explained before. The new variable, in the example, will contain the list of combinations of water access technologies available in each city. This enables also to analyze the kind of combinations really used or not and some analysis on the interactions between water access technologies. Also, some expert-based feature extraction can be done to substitute the multivalued variable by a vector of more powerful variables describing it. In the example, a possibility is building the number of water access technologies available in the city, and, for example, an indicator for the existence of rudimentary systems or most modern ones, etc. This clearly depends on the goal of the analysis and must be carefully designed using specific domain-knowledge. In Section 9.2.3 the factorial methods have been presented as a method to reduce data dimensionality. The retained factors are in fact new variables, which correspond to (usually linear) combinations of the original variables and can be used instead of them in the data mining. It is important to ensure that the factors can be properly interpreted, to get meaningful data mining models. Finally, when serial measurements are obtained, from sensors for example, it is dangerous to introduce the series itself in the analysis as it is, for the same reasons discussed in the multivalued variables case. Imagine a dataset describing some pieces of land, with their surface, characteristics of the soil, climate, proximity to the see, type of culture, agricultural technology used (including type of fertilizers, pesticides, etc.), obtained production, etc., and including a set of 365 variables with the daily mean temperatures along a complete year, another with the minimum daily temperatures, and another with the maximum temperatures. Building data mining models to analyze the relationship between production and the other factors, cannot include directly the 365 temperature variables as they are. In this case, expert-based feature extraction can be used to substitute the temperature series to a subset of meaningful variables describing the series. For example, mean temperature in the period of growing the vegetable, max temperature in the same period, min temperature in the same period, some synthetic descriptors or indicators, as for example, was the temperature below 0 degrees in between November and February?. In BIB001 it is shown that reducing the series to a single mean significantly decreases the quality of the data mining models.