Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VIII. DISCUSSION <s> Increasing the awareness of how incomplete data affects learning and classification accuracy has led to increasing numbers of missing data techniques. This article investigates the robustness and accuracy of seven popular techniques for tolerating incomplete training and test data for different patterns of missing data—different proportions and mechanisms of missing data on resulting tree-based models. The seven missing data techniques were compared by artificially simulating different proportions, patterns, and mechanisms of missing data using 21 complete datasets (i.e., with no missing values) obtained from the University of California, Irvine repository of machine-learning databases (Blake and Merz, 1998). A four-way repeated measures design was employed to analyze the data. The simulation results suggest important differences. All methods have their strengths and weaknesses. However, listwise deletion is substantially inferior to the other six techniques, while multiple imputation, that utilizes the expectation maximization algorithm, represents a superior approach to handling incomplete data. Decision tree single imputation and surrogate variables splitting are more severely impacted by missing values distributed among all attributes compared to when they are only on a single attribute. Otherwise, the imputation—versus model-based imputation procedures gave—reasonably good results although some discrepancies remained. Different techniques for addressing missing values when using decision trees can give substantially diverse results, and must be carefully considered to protect against biases and spurious findings. Multiple imputation should always be used, especially if the data contain many missing values. If few values are missing, any of the missing data techniques might be considered. The choice of technique should be guided by the proportion, pattern, and mechanisms of missing data, especially the latter two. However, the use of older techniques like listwise deletion and mean or mode single imputation is no longer justifiable given the accessibility and ease of use of more advanced techniques, such as multiple imputation and supervised learning imputation. <s> BIB001 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> VIII. DISCUSSION <s> Objectives: Many classification problems must deal with data that contains missing values. In such cases data imputation is critical. This paper evaluates the performance of several statistical and machine learning imputation methods, including our novel multiple imputation ensemble approach, using different datasets. Materials and methods: Several state-of-the-art approaches are compared using different datasets. Some state-of-the-art classifiers (including support vector machines and input decimated ensembles) are tested with several imputation methods. The novel approach proposed in this work is a multiple imputation method based on random subspace, where each missing value is calculated considering a different cluster of the data. We have used a fuzzy clustering approach for the clustering algorithm. Results: Our experiments have shown that the proposed multiple imputation approach based on clustering and a random subspace classifier outperforms several other state-of-the-art approaches. Using the Wilcoxon signed-rank test (reject the null hypothesis, level of significance 0.05) we have shown that the proposed best approach is outperformed by the classifier trained using the original data (i.e., without missing values) only when >20% of the data are missed. Moreover, we have shown that coupling an imputation method with our cluster based imputation we outperform the base method (level of significance ~0.05). Conclusion: Starting from the assumptions that the feature set must be partially redundant and that the redundancy is distributed randomly over the feature set, we have proposed a method that works quite well even when a large percentage of the features is missing (>=30%). Our best approach is available (MATLAB code) at bias.csr.unibo.it/nanni/MI.rar. <s> BIB002
Overall, we may divide the issues of reviewed approaches into three different types: Theoretical flaws, Empirical flaws and Experimental Setup hazards. Theoretical flaws refer to design flaws in the approaches: problems that may arise in some of the key ideas of the approach. Empirical flaws refer to some issues that may occur not (solely) due to the rationale behind each approach, but generated by specific conditions that may arise in some domains (e.g. different feature types), often discussed throughout the paper. Finally, Experimental Setup hazards are not considered flaws inherent to the approaches per se, but refer to some details that should be taken into account: they are considered hazards in the sense that they are risks, but can easily be surpassed by a careful experimental design. • Theoretical flaws: • Usage of Bernoulli trials: For datasets with a small number of patterns (small n), Bernoulli trials may not provide the desired MR. To surpass this issue several algorithms use random permutations of x i,j positions instead. • Definition of pairs/triples and consequent MR adjustments: On one hand, defining pairs/triples of features is an interesting approach since we guarantee that there is a relation between the features. In MAR and MNAR, for each missing feature, there is another highly correlated with it (completely observed) that, in theory, possesses information that may be relevant when imputing the missing values. However, defining these pairs/triples may condition the MR greatly, due to the necessary adjustments: depending on the implementation, the MR may be limited from less than 50% to less that 25%. • Empirical flaws: • Usage of the median to define groups: Using the median generally aggravates the MR restrictions, especially for MAR unifo implementations. Furthermore, if the dataset comprises qualitative features, the use of the median can, in some situations, weaken the mechanisms or fail to provide the specified MR (e.g. MAR3 univa , MAR3 unifo , MAR4 unifo , MNAR3 unifo , MNAR4 unifo , among others). • Usage of cut-off values defined using percentiles: Defining a cut-off value and deleting values accordingly might fail to provide the desired MR, especially if qualitative features are at state, as explained throughout the paper. Among all implementations, cut-off values based on percentiles are only considered in Twala BIB001 , which could be replaced by a sorting of the values to guarantee that the necessary MR is respected. Nevertheless, the sorting requires that a feature can be ordered, which is not always the case. • Experimental Setup hazards: • Random choice of determining and missing features: If we consider a typical experimental setup where n datasets are chosen to generate missing values, one important aspect is to make the experiments as consistent as possible. As an example, consider a dataset X where MAR values are generated in MRs of 10%, 20%, 30% and so on. If missing values are generated according to MAR1 unifo , for instance, where the determining and missing features are randomly chosen, there are several factors (besides the increase of the MR) that affect the final results. These type of assumptions and limitations need to be established apriori, according to the objectives of the experiments. In some cases, the presented domain-based approaches might be worthy of consideration (see BIB002 ), adapting the missing value generation to the context and objectives of the study. • Variability of generated missing datasets: In some cases, especially in MCAR approaches, VOLUME 7, 2019 the possibilities of obtaining different outcomes is enormous and, therefore, several runs should be performed. As an example, two different runs of MAR2 unifo might provide datasets with different ''difficulty degrees'' for imputation algorithms. Nevertheless, this is not an issue of the approach per se but should be bypassed by the design of experiments.