Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Effort prediction is a very important issue for software project management. Historical project data sets are frequently used to support such prediction. But missing data are often contained in these data sets and this makes prediction more difficult. One common practice is to ignore the cases with missing data, but this makes the originally small software project database even smaller and can further decrease the accuracy of prediction. The alternative is missing data imputation. There are many imputation methods. Software data sets are frequently characterised by their small size but unfortunately sophisticated imputation methods prefer larger data sets. For this reason we explore using simple methods to impute missing data in small project effort data sets. We propose a class mean imputation (CMI) method based on the k-NN hot deck imputation method (MINI) to impute both continuous and nominal missing data in small data sets. We use an incremental approach to increase the variance of population. To evaluate MINI (and k-NN and CMI methods as benchmarks) we use data sets with 50 cases and 100 cases sampled from a larger industrial data set with 10%, 15%, 20% and 30% missing data percentages respectively. We also simulate Missing Completely at Random (MCAR) and Missing at Random (MAR) missingness mechanisms. The results suggest that the MINI method outperforms both CMI and the k-NN methods. We conclude that this new imputation technique can be used to impute missing values in small data sets. <s> BIB001 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Objectives: Many classification problems must deal with data that contains missing values. In such cases data imputation is critical. This paper evaluates the performance of several statistical and machine learning imputation methods, including our novel multiple imputation ensemble approach, using different datasets. Materials and methods: Several state-of-the-art approaches are compared using different datasets. Some state-of-the-art classifiers (including support vector machines and input decimated ensembles) are tested with several imputation methods. The novel approach proposed in this work is a multiple imputation method based on random subspace, where each missing value is calculated considering a different cluster of the data. We have used a fuzzy clustering approach for the clustering algorithm. Results: Our experiments have shown that the proposed multiple imputation approach based on clustering and a random subspace classifier outperforms several other state-of-the-art approaches. Using the Wilcoxon signed-rank test (reject the null hypothesis, level of significance 0.05) we have shown that the proposed best approach is outperformed by the classifier trained using the original data (i.e., without missing values) only when >20% of the data are missed. Moreover, we have shown that coupling an imputation method with our cluster based imputation we outperform the base method (level of significance ~0.05). Conclusion: Starting from the assumptions that the feature set must be partially redundant and that the redundancy is distributed randomly over the feature set, we have proposed a method that works quite well even when a large percentage of the features is missing (>=30%). Our best approach is available (MATLAB code) at bias.csr.unibo.it/nanni/MI.rar. <s> BIB002 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> OBJECTIVE ::: To understand how handling of missing data influences the statistical power and bias of treatment effects in randomised controlled trials of painful knee osteoarthritis (OA). ::: ::: ::: METHODS ::: We simulated trials with missing data (withdrawals) due to lack-of-efficacy. Outcome measures were response/non-response according to the Outcome Measures in Rheumatology-Osteoarthritis Research Society International (OMERACT-OARSI) set of responder criteria, Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain and physical function from the WOMAC questionnaire, and patient global assessment. We used five methods for managing missing data: ignoring the missing data, last and baseline observation carried forward (LOCF and BOCF), and multiple imputation with two different strategies. The treatment effect was then analysed by appropriate univariate and longitudinal statistical methods, and power, bias and mean squared error (MSE) was assessed by comparing the estimated treatment effect in the trials with missing data with the estimated treatment effect on the trials without missing data. ::: ::: ::: RESULTS ::: The best imputation method in terms of high power and low bias/MSE was our implementation of regression multiple imputation. The most conservative method was the data augmentation Markov chain Monte Carlo (MCMC) multiple imputation. The LOCF, BOCF and the complete-case methods were not particularly conservative and gave relatively low power and high bias. The analysis on the WOMAC pain scale gave less bias and higher power than the OMERACT-OARSI responder outcome measure. ::: ::: ::: CONCLUSIONS ::: Multiple imputation of missing data may be used to decrease bias/MSE and increase power in OA trials. These results can guide investigators in the choice of outcome measures and especially how missing data can be handled. <s> BIB003 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Missing covariate data is a common problem in nonlinear mixed effects modelling of clinical data. The aim of this study was to implement and compare methods for handling missing covariate data in nonlinear mixed effects modelling under different missing data mechanisms. Simulations generated data for 200 individuals with a 50% difference in clearance between males and females. Three different types of missing data mechanisms were simulated and information about sex was missing for 50% of the individuals. Six methods for handling the missing covariate were compared in a stochastic simulations and estimations study where 200 data sets were simulated. The methods were compared according to bias and precision of parameter estimates. Multiple imputation based on weight and response, full maximum likelihood modelling using information on weight and full maximum likelihood modelling where the proportion of males among the individuals lacking information about sex was estimated (EST) gave precise and unbiased estimates in the presence of missing data when data were missing completely at random or missing at random. When data were missing not at random, the only method resulting in low bias and high parameter precision was EST. <s> BIB004 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> We present two novel techniques for the imputation of both categorical and numerical missing values. The techniques use decision trees and forests to identify horizontal segments of a data set where the records belonging to a segment have higher similarity and attribute correlations. Using the similarity and correlations, missing values are then imputed. To achieve a higher quality of imputation some segments are merged together using a novel approach. We use nine publicly available data sets to experimentally compare our techniques with a few existing ones in terms of four commonly used evaluation criteria. The experimental results indicate a clear superiority of our techniques based on statistical analyses such as confidence interval. <s> BIB005 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Natural data sets often have missing values in them. An accurate missing value imputation is crucial to increase the usability of a data set for statistical analyses and data mining tasks. In this paper we present a novel missing value imputation technique using a data set's existing patterns including co-appearances of attribute values, correlations among the attributes and similarity of values belonging to an attribute. Our technique can impute both numerical and categorical missing values. We carry out extensive experiments on nine natural data sets, and compare our technique with four high quality existing techniques. We simulate 32 types of missing patterns (combinations), and thereby generate 320 missing data sets for each of the nine natural data sets. Two well known evaluation criteria namely index of agreement (d"2) and root mean squared error are used. Our experimental results, based on the statistical sign test, indicate that our technique achieves significantly better imputation accuracy than the existing techniques. <s> BIB006 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> The goal is to investigate the prediction performance of tree-based techniques when the available training data contains features with missing values. Also the future test cases may contain missing values and thus the methods should be able to generate predictions for such test cases. The missing values are handled either by using surrogate decisions within the trees or by the combination of an imputation method with a tree-based method. Missing values generated according to missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR) mechanisms are considered with various fractions of missing data. Imputation models are built in the learning phase and do not make use of the response variable, so that the resulting procedures allow to predict individual incomplete test cases. In the empirical comparison, both classification and regression problems are considered using a simulated and real-life datasets. The performance is evaluated by misclassification rate of predictions and mean squared prediction error, respectively. Overall, our results show that for smaller fractions of missing data an ensemble method combined with surrogates or single imputation suffices. For moderate to large fractions of missing values ensemble methods based on conditional inference trees combined with multiple imputation show the best performance, while conditional bagging using surrogates is a good alternative for high-dimensional prediction problems. Theoretical results confirm the potential better prediction performance of multiple imputation ensembles. <s> BIB007 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Death, injury and disability resulting from road traffic crashes continue to be a major global public health problem. Recent data suggest that the number of fatalities from traffic crashes is in excess of 1.25 million people each year with non-fatal injuries affecting a further 20-50 million people. It is predicted that by 2030 road traffic accidents will have progressed to be the 5th leading cause of death and that the number of people who will die annually from traffic accidents will have doubled from current levels. Both developed and developing countries suffer from the consequences of increase in human population, and therefore, vehicle population. Therefore, methods to reduce accident severity are of great interest to traffic agencies and the public at large. To analyse traffic accident factors effectively we need a complete traffic accident historical database. Any missing data in the database could prevent the discovery of important environmental and road accident factors and lead to invalid conclusions. In this paper, we present a novel imputation method that exploits the within-record and between-record correlations to impute missing data of numerical or categorical values. In addition, our algorithm accounts for uncertainty in real world data by sampling from a list of potential imputed values according to their affinity degree. We evaluated our algorithm using four publicly available traffic accident databases from the United States, the first of which is the largest open federal database (explore.data.gov) in the United States, and the second is based on the National Incident Based Reporting System (NIBRS) of the city and county of Denver (data.opencolorado.org). The other two are from New York's open data portal (Motor Vehicle Crashes-case information: 2011 and Motor Vehicle Crashes-individual information: 2011, data.ny.gov). We compare our algorithm with four state-of-the-art imputation methods using missing value imputation accuracy and RMSE. Our results indicate that the proposed method performs significantly better than the existing algorithms we compared. <s> BIB008 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VII. DOMAIN-BASED MISSING DATA GENERATION APPROACHES <s> Abstract When applying data-mining techniques to real-world data, we often find ourselves facing observations that have no value recorded for some attributes. This can be caused by several phenomena, such as a machine’s incapability to record certain characteristics or a person refusing to answer a question in a poll. Depending on that motivation, values gone missing may follow one kind of pattern or another, or describe no regularity at all. One approach to palliate the effect of missing data on machine learning tasks is to replace the missing observations. Imputation algorithms attempt to calculate a value for a missing gap, using information associated with it, i.e., the attribute and/or other values in the same observation. While several imputation methods have been proposed in the literature, few works have addressed the question of the relationship between the type of missing data, the choice of the imputation method, and the effectiveness of classification algorithms that used the imputed data. In this paper we address the relationship among these three factors. By constructing a benchmark of hundreds of databases containing different types of missing data, and applying several imputation methods and classification algorithms, we empirically show that an interaction between imputation methods and supervised classification can be deduced. Besides, differences in terms of classification performance for the same imputation method in different missing data patterns have been found. This points to the convenience of considering the combined choice of the imputation method and the classifier algorithm according to the missing data type. <s> BIB009
The implementations presented in the previous sections are rather generic approaches to missing data generation. They were developed for general domains, with no particular focus on the peculiarities of a given domain and without assuming any apriori knowledge of the domain (e.g. known relationships between features in the study). However, some missing data generation approaches found in the literature are adapted to the domain in question. In this section, we review some domain-specific approaches to missing data generation. Some, although uncommon, may be generalized to different domains; others are not generalizable but may contain interesting details to consider for some real-world domains (e.g. healthcare domains). Song and Shepperd BIB001 focus on evaluating imputation methods for small project effort data sets. In this domain, MAR data is generated according to the size of the project. First, records are ordered by project size; then, the dataset is divided into 4 parts with different percentages of missing data: for each part d, its missing percentage is proportional to Josse et al.  use synthetic data to generate two different MAR scenarios, ''MAR easy'' and ''MAR difficult'' for a simulated dataset comprising 9 features that could be divided into two blocks of correlated features: {x 1 , x 2 , x 3 , x 4 , x 5 } and {x 6 , x 7 , x 8 , x 9 }. Then, ''MAR easy'' would consist of deleting values of x 2 to x 5 according to values of x 1 and deleting FIGURE 11. Missing data pattern of the MCAR implementation by Nanni et al. BIB002 . values of x 6 to x 8 according to values of x 9 . This traduces a situation where the missing values are easier to recover given the known existing correlation between features. ''MAR difficult'' worked by deleting values of x 6 to x 9 according to values of x 1 and deleting values of x 1 to x 5 according to values of x 9 , so that the available information to predict missing values is very limited. Johansson and Karlsson BIB004 focus on strategies to handle missing values in clinical data. A pharmacokinetic model was used to generate a synthetic dataset where missing values were generated in feature ''Sex''. For MCAR, values of ''Sex'' were randomly deleted; for MAR missing values in ''Sex'' were generated according to the ''Weight'' of the subjects and finally, for MNAR, missing values in ''Sex'' were deleted for male subjects. Olsen et al. BIB003 study the effects of handling missing data in clinical trials of knee osteoarthritis. Missing data was generated in two MNAR scenarios: Scenario A, where the probability of missing data was dependent on changes of pain, physical function and patient's global assessment, and Scenario B, where the missingness was dependent on type of treatment and consequent effects. Nanni et al. BIB002 focus on discovering an imputation method that would perform well in medical domains. Thus, authors generate MCAR data in a different fashion: instead of generating MR% of missing values in each feature or in the whole dataset directly (deleting MR% of x i,j elements), the missing values are generated in each pattern x i . In other words, each pattern x i will have MR% of missing values, where different features can be missing for different patterns ( Figure 11 ). This translates a context where all patients have at least one missing observation. Deb and Liew BIB008 study missing value imputation for the analysis of traffic accident data and generate missing values in a similar way to Nanni et al. BIB002 : missing values are generated by pattern, rather than by feature. This generation method follows from the research of Rahman and Islam BIB005 , BIB006 and considers four main configuration types: Simple, Medium, Complex and Blended. In the Simple generation, each pattern x i has at most one missing value; in Medium generation, each pattern x i has a minimum of 2 missing values and at most 50% missing values and in Complex generation, each pattern x i has between 51% and 80% missing values. Finally, Blended generation considers a mixture of the remaining types -25%, 50% and 25% of patterns according to Simple, Medium and Complex generation types, respectively. Furthermore, two different models for missing data generation are used: Uniformly Distributed (UD) and Overall models. In the UD model, it is guaranteed that all features have the same amount of missing values, whereas in the Overall model missing values can be dispersed across several features (in the worst-case scenario, they can all appear in a single feature). Garciarena and Santana BIB009 , as mentioned in Section V-C, propose another version to generate MNAR data, called MuOv (Missing depending on unobserved Variables). MuOv represents a MNAR scenario where the probability of missing values in a feature is related to some other feature that was not considered in the study. In this case, N patterns are randomly chosen to be missing (according to the desired MR) and their values on each feature to be missing are deleted (Figure 12 ). Although MuOv does not consider the application to a specific domain, we have included it here since it is rather an uncommon MNAR approach, as previously discussed. Valdiviezo and Van Aelst BIB007 introduced missing values in real-world datasets according to different mechanisms and schemes. In general, two schemes are followed for each mechanism: either considering all features (first scheme) or considering only one-third of features, which are randomly chosen (second scheme). Regarding MCAR mechanism, the first scheme inserts MR% of missing values in each feature while in the second scheme, since only one-third of features will have missing data, each of those features will have 3 × MR% of missing values. In the original paper, this adjustment is not mentioned, but we have decided to discuss it so that the overall MR is respected. In MAR mechanism, the first scheme randomly selects one feature to be the determining feature, x obs , and the remaining p − 1 features will have their values missing according to the values of x obs . To that end, the values of x obs are transformed into probabilities by a logistic function, and the missing locations for the remaining features are sampled according to such probability. Finally, in MNAR mechanism, the first scheme deletes the highest or lowest values of each feature in the dataset, while the second scheme proceeds in the same way but only for onethird of features. Soares et al.  study how different methods behave when imputing data from different continuous distributions. To that end, each feature is fitted against a comprehensive set of continuous distributions and missing values are generated according to 7 distinct methods, T 1 to T 7 . Method T 7 is a standard MCAR approach (MCAR2 univa ), where the same amount of missing values are randomly inserted in each feature. The remaining methods are MNAR approaches, where the missing values are removed according to each feature's probability density functions or frequency histograms. Methods T 1 to T 3 are pdf-based while methods T 4 to T 6 are freq-based. For each method, three different scenarios are considered: removing from the outer areas, inner areas or both. Outer and inner areas correspond to low and high values of the pdf and frequency histogram, respectively ( Figure 13 ).