Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Effort prediction is a very important issue for software project management. Historical project data sets are frequently used to support such prediction. But missing data are often contained in these data sets and this makes prediction more difficult. One common practice is to ignore the cases with missing data, but this makes the originally small software project database even smaller and can further decrease the accuracy of prediction. The alternative is missing data imputation. There are many imputation methods. Software data sets are frequently characterised by their small size but unfortunately sophisticated imputation methods prefer larger data sets. For this reason we explore using simple methods to impute missing data in small project effort data sets. We propose a class mean imputation (CMI) method based on the k-NN hot deck imputation method (MINI) to impute both continuous and nominal missing data in small data sets. We use an incremental approach to increase the variance of population. To evaluate MINI (and k-NN and CMI methods as benchmarks) we use data sets with 50 cases and 100 cases sampled from a larger industrial data set with 10%, 15%, 20% and 30% missing data percentages respectively. We also simulate Missing Completely at Random (MCAR) and Missing at Random (MAR) missingness mechanisms. The results suggest that the MINI method outperforms both CMI and the k-NN methods. We conclude that this new imputation technique can be used to impute missing values in small data sets. <s> BIB001 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Missing data imputation is an important research topic in data mining. The impact of noise is seldom considered in previous works while real-world data often contain much noise. In this paper, we systematically investigate the impact of noise on imputation methods and propose a new imputation approach by introducing the mechanism of Group Method of Data Handling (GMDH) to deal with incomplete data with noise. The performance of four commonly used imputation methods is compared with ours, called RIBG (robust imputation based on GMDH), on nine benchmark datasets. The experimental result demonstrates that noise has a great impact on the effectiveness of imputation techniques and our method RIBG is more robust to noise than the other four imputation methods used as benchmark. <s> BIB002 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> OBJECTIVE ::: To understand how handling of missing data influences the statistical power and bias of treatment effects in randomised controlled trials of painful knee osteoarthritis (OA). ::: ::: ::: METHODS ::: We simulated trials with missing data (withdrawals) due to lack-of-efficacy. Outcome measures were response/non-response according to the Outcome Measures in Rheumatology-Osteoarthritis Research Society International (OMERACT-OARSI) set of responder criteria, Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain and physical function from the WOMAC questionnaire, and patient global assessment. We used five methods for managing missing data: ignoring the missing data, last and baseline observation carried forward (LOCF and BOCF), and multiple imputation with two different strategies. The treatment effect was then analysed by appropriate univariate and longitudinal statistical methods, and power, bias and mean squared error (MSE) was assessed by comparing the estimated treatment effect in the trials with missing data with the estimated treatment effect on the trials without missing data. ::: ::: ::: RESULTS ::: The best imputation method in terms of high power and low bias/MSE was our implementation of regression multiple imputation. The most conservative method was the data augmentation Markov chain Monte Carlo (MCMC) multiple imputation. The LOCF, BOCF and the complete-case methods were not particularly conservative and gave relatively low power and high bias. The analysis on the WOMAC pain scale gave less bias and higher power than the OMERACT-OARSI responder outcome measure. ::: ::: ::: CONCLUSIONS ::: Multiple imputation of missing data may be used to decrease bias/MSE and increase power in OA trials. These results can guide investigators in the choice of outcome measures and especially how missing data can be handled. <s> BIB003 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Objectives: Many classification problems must deal with data that contains missing values. In such cases data imputation is critical. This paper evaluates the performance of several statistical and machine learning imputation methods, including our novel multiple imputation ensemble approach, using different datasets. Materials and methods: Several state-of-the-art approaches are compared using different datasets. Some state-of-the-art classifiers (including support vector machines and input decimated ensembles) are tested with several imputation methods. The novel approach proposed in this work is a multiple imputation method based on random subspace, where each missing value is calculated considering a different cluster of the data. We have used a fuzzy clustering approach for the clustering algorithm. Results: Our experiments have shown that the proposed multiple imputation approach based on clustering and a random subspace classifier outperforms several other state-of-the-art approaches. Using the Wilcoxon signed-rank test (reject the null hypothesis, level of significance 0.05) we have shown that the proposed best approach is outperformed by the classifier trained using the original data (i.e., without missing values) only when >20% of the data are missed. Moreover, we have shown that coupling an imputation method with our cluster based imputation we outperform the base method (level of significance ~0.05). Conclusion: Starting from the assumptions that the feature set must be partially redundant and that the redundancy is distributed randomly over the feature set, we have proposed a method that works quite well even when a large percentage of the features is missing (>=30%). Our best approach is available (MATLAB code) at bias.csr.unibo.it/nanni/MI.rar. <s> BIB004 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Missing covariate data is a common problem in nonlinear mixed effects modelling of clinical data. The aim of this study was to implement and compare methods for handling missing covariate data in nonlinear mixed effects modelling under different missing data mechanisms. Simulations generated data for 200 individuals with a 50% difference in clearance between males and females. Three different types of missing data mechanisms were simulated and information about sex was missing for 50% of the individuals. Six methods for handling the missing covariate were compared in a stochastic simulations and estimations study where 200 data sets were simulated. The methods were compared according to bias and precision of parameter estimates. Multiple imputation based on weight and response, full maximum likelihood modelling using information on weight and full maximum likelihood modelling where the proportion of males among the individuals lacking information about sex was estimated (EST) gave precise and unbiased estimates in the presence of missing data when data were missing completely at random or missing at random. When data were missing not at random, the only method resulting in low bias and high parameter precision was EST. <s> BIB005 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Display Omitted Survival prediction of HCC patients is modeled in a real and high complex scenario.Its complexity is due to a small dataset with heterogeneous and missing input data.We propose a new cluster-based oversampling method to overcome these drawbacks.Neural networks and logistic regression are applied in the classification stage.Experimental results show that our method outperforms previous existing approaches. Liver cancer is the sixth most frequently diagnosed cancer and, particularly, Hepatocellular Carcinoma (HCC) represents more than 90% of primary liver cancers. Clinicians assess each patients treatment on the basis of evidence-based medicine, which may not always apply to a specific patient, given the biological variability among individuals. Over the years, and for the particular case of Hepatocellular Carcinoma, some research studies have been developing strategies for assisting clinicians in decision making, using computational methods (e.g. machine learning techniques) to extract knowledge from the clinical data. However, these studies have some limitations that have not yet been addressed: some do not focus entirely on Hepatocellular Carcinoma patients, others have strict application boundaries, and none considers the heterogeneity between patients nor the presence of missing data, a common drawback in healthcare contexts. In this work, a real complex Hepatocellular Carcinoma database composed of heterogeneous clinical features is studied. We propose a new cluster-based oversampling approach robust to small and imbalanced datasets, which accounts for the heterogeneity of patients with Hepatocellular Carcinoma. The preprocessing procedures of this work are based on data imputation considering appropriate distance metrics for both heterogeneous and missing data (HEOM) and clustering studies to assess the underlying patient groups in the studied dataset (K-means). The final approach is applied in order to diminish the impact of underlying patient profiles with reduced sizes on survival prediction. It is based on K-means clustering and the SMOTE algorithm to build a representative dataset and use it as training example for different machine learning procedures (logistic regression and neural networks). The results are evaluated in terms of survival prediction and compared across baseline approaches that do not consider clustering and/or oversampling using the Friedman rank test. Our proposed methodology coupled with neural networks outperformed all others, suggesting an improvement over the classical approaches currently used in Hepatocellular Carcinoma prediction models. <s> BIB006 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Treatment of missing data has become increasingly significant in scientific research and engineering applications. The classic imputation strategy based on the K nearest neighbours (KNN) has been widely used to solve the plague problem. However, former studies do not give much attention to feature relevance, which has a significant impact on the selection of nearest neighbours. As a result, biased results may appear in similarity measurements. In this paper, we propose a novel method to impute missing data, named feature weighted grey KNN (FWGKNN) imputation algorithm. This approach employs mutual information (MI) to measure feature relevance. We present an experimental evaluation for five UCI datasets in three missingness mechanisms with various missing rates. Experimental results show that feature relevance has a non-ignorable influence on missing data estimation based on grey theory, and our method is considered superior to the other four estimation strategies. Moreover, the classification bias can be significantly reduced by using our approach in classification tasks. <s> BIB007 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Background: Recurrence is an important cornerstone in breast cancer behavior, intrinsically related to mortality. In spite of its relevance, it is rarely recorded in the majority of breast cancer datasets, which makes research in its prediction more difficult. Objectives: To evaluate the performance of machine learning techniques applied to the prediction of breast cancer recurrence. Material and Methods: Revision of published works that used machine learning techniques in local and open source databases between 1997 and 2014. Results: The revision showed that it is difficult to obtain a representative dataset for breast cancer recurrence and there is no consensus on the best set of predictors for this disease. High accuracy results are often achieved, yet compromising sensitivity. The missing data and class imbalance problems are rarely addressed and most often the chosen performance metrics are inappropriate for the context. Discussion and Conclusions: Although different techniques have been used, prediction of breast cancer recurrence is still an open problem. The combination of different machine learning techniques, along with the definition of standard predictors for breast cancer recurrence seem to be the main future directions to obtain better results. <s> BIB008 </s> Generating Synthetic Missing Data: A Review by Missing Mechanism <s> IX. CONCLUSIONS AND POTENTIAL RESEARCH DIRECTIONS <s> Death, injury and disability resulting from road traffic crashes continue to be a major global public health problem. Recent data suggest that the number of fatalities from traffic crashes is in excess of 1.25 million people each year with non-fatal injuries affecting a further 20-50 million people. It is predicted that by 2030 road traffic accidents will have progressed to be the 5th leading cause of death and that the number of people who will die annually from traffic accidents will have doubled from current levels. Both developed and developing countries suffer from the consequences of increase in human population, and therefore, vehicle population. Therefore, methods to reduce accident severity are of great interest to traffic agencies and the public at large. To analyse traffic accident factors effectively we need a complete traffic accident historical database. Any missing data in the database could prevent the discovery of important environmental and road accident factors and lead to invalid conclusions. In this paper, we present a novel imputation method that exploits the within-record and between-record correlations to impute missing data of numerical or categorical values. In addition, our algorithm accounts for uncertainty in real world data by sampling from a list of potential imputed values according to their affinity degree. We evaluated our algorithm using four publicly available traffic accident databases from the United States, the first of which is the largest open federal database (explore.data.gov) in the United States, and the second is based on the National Incident Based Reporting System (NIBRS) of the city and county of Denver (data.opencolorado.org). The other two are from New York's open data portal (Motor Vehicle Crashes-case information: 2011 and Motor Vehicle Crashes-individual information: 2011, data.ny.gov). We compare our algorithm with four state-of-the-art imputation methods using missing value imputation accuracy and RMSE. Our results indicate that the proposed method performs significantly better than the existing algorithms we compared. <s> BIB009
This manuscript reviews a considerable number of missing data generation approaches, for different configurations (univariate and multivariate) and missing data mechanisms (MCAR, MAR, and MNAR). Their limitations are discussed from a theoretical and empirical view and some modifications are suggested in order to surpass them. Additionally, we refer some less common approaches -herein named ''domainbased'' approaches -in order to illustrate existing missing data generation approaches in specific contexts. The theoretical flaws may compromise/constraint the possible MRs to generate; nevertheless, this problem is easy to diagnose and, although the desired percentage of missing values may not be achieved, there is no risk of breaking the assumptions regarding the missing mechanisms. Regarding the identified empirical flaws, it is important to state that they are mostly related to the existence of qualitative features with no order (nominal features), which is very common across several domains BIB006 , BIB008 . This is the most challenging topic to solve in related work and is most often neglected. With the exception of Zhu et al. BIB002 and Pan et al. BIB007 , which distinguish between ordered and nominal features, no other work refers to this issue. This limitation becomes more evident when using the median or percentiles/quantiles, which require that the features have an order, although in any implementation that requires values to be ordered (independently of the use of median or percentiles), this problem exists. These empirical flaws are more serious since they may bias the missing mechanism. The experimental setup hazards are unrelated to the described approaches, but they might be induced inadvertently by the researcher during the study. Therefore, they will not affect certain aspects of the implementation (faulty MR rate, broken missing mechanism), but they may compromise the derived insights for certain implementations, if there is not a careful experimental design (e.g overlooking the stochastic process inherent to the MCAR unifo approaches). Domain-based approaches are mainly developed in order to adapt to given contexts: they arise when there is a need to study specific situations/properties in data (Josse et al.  and Soares et al.  ) or to map known relationships in data (Johansson and Karlsson BIB005 and Olsen et al. BIB003 ), or they may also reflect the reality of certain domains, such as healthcare domains (Nanni et al. BIB004 ), software management (Song and Shepperd BIB001 ) and traffic data (Deb and Liew BIB009 ). A standard approach in this case is to generate missing values per pattern, rather than per feature. This is a way of traducing the reality in these domains: as an example, in medical datasets, it is not expected that certain features are absent for all patients, but instead, that several patients have absent observations in some features BIB006 . Although some of these approaches are not generalizable, we have decided to present them since they represent valid approaches in certain contexts and might inspire other approaches for similar domains. Finally, we shall refer to some potential research directions in the field: • Generating MAR and MNAR with nominal features: The definition of appropriate strategies to generate MAR and MNAR data with nominal features would be important, since most strategies proposed so far may fail under certain circumstances; • Generating MAR through data modeling: In the reviewed works, MAR either makes use of one determining feature, or pairs of features where one in the pair is the determining feature. Future research could explore the effects of generating MAR via the combination of all features in data (except for the missing features); • Investing in software development: Nowadays, a great number of statistical software (SPSS, R, MatLab) considers the development of models with missing data and procedures for MD imputation. Nevertheless, strategies for MD generation are most often neglected; • Experimenting over real-world datasets: Investigating the reliability and consistency of the methods outlined in this work on a large benchmark of real-work datasets ( MIRIAM SEOANE SANTOS received the master's degree in biomedical engineering from the University of Coimbra, in 2014, where she is currently pursuing the Ph.D. degree in information science and technology. She is also a member of the Centre of Informatics and Systems, University of Coimbra. Her research interests include pattern recognition problems, imbalanced and missing data, and personalized medicine in oncology. RICARDO CARDOSO PEREIRA received the bachelor's degree from the Coimbra Institute of Engineering and the master's degree in informatics engineering from the University of Coimbra, where he is currently pursuing the Ph.D. degree. He is also an Invited Assistant Teacher with the University of Coimbra. He also teaches at the Coimbra Institute of Engineering. His main research interests include computational intelligence and data processing, with a special emphasis on the missing data field. ADRIANA FONSECA COSTA received the master's degree in biomedical engineering from the University of Coimbra, in 2018. She specialized in clinical informatics and bioinformatics. Her current research interests include data preprocessing problems, such as missing data, and deep learning applications to data imputation. PEDRO HENRIQUES ABREU received the degree in informatics engineering and the Ph.D. degree in soccer teams modeling from the University of Porto, in 2006 and 2011, respectively. He is currently an Assistant Professor with the Department of Informatics Engineering, University of Coimbra. He has authored more than 60 publications in international conferences and journals. His research interests include medical informatics and personal healthcare systems applied to oncology. VOLUME 7, 2019 