An in-depth performance analysis of the oversampling techniques for high-class imbalanced dataset

Article history: Received 23 December 2020 Revised 13 February 2021 Accepted 18 February 2021 Available online 28 February 2021 Class imbalance occurs when the distribution of classes between the majority and the minority classes is not the same. The data on imbalanced classes may vary from mild to severe. The effect of highclass imbalance may affect the overall classification accuracy since the model is most likely to predict most of the data that fall within the majority class. Such a model will give biased results, and the performance predictions for the minority class often have no impact on the model. The use of the oversampling technique is one way to deal with high-class imbalance, but only a few are used to solve data imbalance. This study aims for an in-depth performance analysis of the oversampling techniques to address the high-class imbalance problem. The addition of the oversampling technique will balance each class’s data to provide unbiased evaluation results in modeling. We compared the performance of Random Oversampling (ROS), ADASYN, SMOTE, and Borderline-SMOTE techniques. All oversampling techniques will be combined with machine learning methods such as Random Forest, Logistic Regression, and k-Nearest Neighbor (KNN). The test results show that Random Forest with Borderline-SMOTE gives the best value with an accuracy value of 0.9997, 0.9474 precision, 0.8571 recall, 0.9000 F1-score, 0.9388 ROCAUC, and 0.8581 PRAUC of the overall oversampling technique.


Introduction
Class imbalance occurs when the distribution of classes between the majority and the minority classes is not equal. In real-world applications, the class imbalance may vary from minor to severe [1]. The dataset can be said to be unbalanced if each class is not represented in a balanced manner. The majority class consists of a large portion of the dataset, while the minority class consists of a small portion of the dataset, which is often the object to be modeled. The consequence of high-class imbalance will impact the classification's overall performance when the model is most likely to predict most of the data falling within the majority class. This model will yield biased results, and the performance predictions for the minority class frequently have little impact on the model. The research conducted by He and Garcia [2] explains that unbalanced data can be seen based on the majority and minority class ratios ranging from 100:1 to 10,000:1. The creation of a model using a high level of imbalance is considered challenging by experts due to the complexity of the processing of balancing data [3].
There are two methods commonly used in modeling data imbalances, namely supervised and semi-supervised [4]. The model will estimate based on the dataset's existing sample to be classified in the supervised approach. Several algorithms can be used, such as Random Forest (RF) [4,5], K-Nearest Neighbors (KNN) [6,7], and Support Vector Machine (SVM) [8,9]. In the semi-supervised approach, data classification will be identified using label and unlabeled data in the dataset. One of the algorithms included in the semi-supervised approach is K-Means [10,11]. Modeling in the high-class data imbalance is important because not all models can provide the actual classification. The unbalanced characteristics of the dataset will result in bias in the model so that a process at the data level is required, such as resampling minority data.
Resampling is a process that tries to balance the distribution of data between minority data and majority data. There are two groups in resampling, namely under-sampling and oversampling. Undersampling is a way to remove or reduce existing samples in the data majority, and oversampling tries to flatten data distribution by duplicating minority data. Oversampling is the focus of this research because all data in the dataset is crucial to be modeled by machine learning.
Currently, the state-of-the-art oversampling technique developed by Chawla [12] is Synthetic Minority Oversampling Technique (SMOTE). This method produces a sample based on the closest rivalry by duplicating it along the lines between minority classes. There are several developments in the oversampling technique method based on the SMOTE method. Each method development has its characteristics, such as Borderline-SMOTE [13], which focuses on duplication on the margins of the minority class, and Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN) [14], which calculates based on the density of minority data. Even though many oversampling techniques have been developed, only a few have been applied to high-class data imbalances.
Many imbalanced datasets can be accessed publicly, such as wine quality [15], cancer diagnosis [16], and credit card fraud detection [17]. The wine quality dataset contains data for classifying the quality of red wine based on wine quality classes. This dataset has 1,599 data with the percentage ratio of minority/majority data of 3.31%/96.69%, which indicates that only about 53 minority data are available in this dataset. Furthermore, cancer diagnosis data contains data to classify whether the patient has cancer or not. This dataset has a total of 1,056 data with a minority/majority data percentage ratio of 18.09%/81.9%, which indicates that there are about 191 minority data that can be modeled. The last dataset is the credit card fraud dataset, which contains data to detect credit card fraud. This dataset's total data is 284,807, with a percentage ratio of the minority/majority data of 0.17%/99.83%. The ratio data shown in the credit card fraud dataset indicates that this dataset experiences a high-class imbalanced dataset with a minority data of 492 from the existing data population. The credit card fraud dataset will be selected in this study because it has the characteristics of a high-class imbalanced dataset.
Several studies use this high-class data imbalance dataset, such as Makki [18], which performed fraud detection using several machine learning models, namely Support Vector Machine and Artificial Neural Network with random oversampling technique. Meng [19] used a combination of XGBoost with oversampling and under-sampling techniques to detect credit card fraud. The obtained result is that the SMOTE technique gives a better score than the under-sampling technique. Almhaithawi [20] used several machine learning methods such as Logistic Regression, Random Forest, and XGBoost by calculating the cost-sensitive Minimum Bayes Risk, which is balanced using SMOTE, which shows good results. Awoyemi [21] used a hybrid technique of under-sampling and oversampling to solve the data imbalance problem. Several machine learning methods used in this test, namely Naïve Bayes, K-NN, and Logistic Regression, showed quite good results. However, all of the above studies use SMOTE as the main oversampling, and a few use other oversampling techniques to solve the problem of high-class data imbalance.
This study aims for an in-depth performance analysis of the oversampling techniques to address the high-class imbalance problem. We compare the performance of Random Oversampling (ROS), ADASYN, SMOTE, and Borderline-SMOTE techniques. All oversampling techniques are combined with machine learning methods such as Random Forest, Logistic Regression, and K-NN. Utilizing the oversampling technique will balance each class's data to provide unbiased evaluation results in modeling. Each characteristic of the oversampling technique has its respective advantages, which will be explained in this study.

Research diagram system
Fig. 1 shows the diagram system used in this study. The initial workflow in the system model is to split up the data using the hold-out method so that it becomes two parts, namely training set, and test set. In the training set, oversampling techniques are used to increase the number of minority class so that when a model fitting is carried out, a balanced dataset can be obtained.  • Random oversampling method (ROS) performs duplication by selecting a random set of minority classes for random data replication [22]. Because the sampling process is carried out randomly, random oversampling has a disadvantage, namely that it requires a long training time, and there is the possibility of overfitting. • SMOTE duplicates data by measuring the similarity between neighboring minority class samples [12]. Each data will be reproduced based on the nearest neighbor line. • Borderline-SMOTE duplicates data by making the closest neighbor line to the sample data on the margins of the minority class [13]. Duplication on this border will strengthen the difference between the minority and majority class. • ADASYN calculates the density distribution in minority classes before duplicating data based on the criteria obtained when calculating class density [14]. This approach helps to focus on the midpoint of the minority class area depending on the dataset.

Description of the dataset
The dataset used in this study is obtained from European cardholder transactions for two days in September 2013 [17]. The dataset is collected by two research teams from Worldline and Libre Brussels University. The dataset contains 30 attributes containing transaction time, number of transactions, and 28 other attributes labeled V1 to V28 through the transformation results using Principal Component Analysis (PCA). This transformation is performed to hide sensitive attribute data from the user. Fig. 3 shows the distribution of the minority and majority classes. There were 284,807 transactions with a positive fraud class percentage of 0.17% and a negative fraud class of 99.83% of the total transactions. Credit card transactions contain numeric and categorical data so that this combination will be processed to detect credit card fraud.

Evaluation metrics
There are four basic metrics used for the evaluation of this study, namely the number of true positive (TP), false positive (FP), false negative (FN), and true negative (TN). TP and TN is the number of samples from the test set that are classified correctly and incorrectly, respectively. In contrast, FN and TP represent the number of samples from the test set, classified incorrectly as negative and positive. A commonly used evaluation metric uses accuracy to get the ratio of how many samples of data were correctly classified. The accuracy is shown in Eq. 1. The information provided by accuracy is sometimes useless when measuring how reliable a model using imbalanced data is. So, in this case, we can measure it using the evaluation metric F1-score. F1-score evaluation requires two metric evaluations: precision and recall, which can be seen in Eq. 2 and Eq. 3.
The F1-score will balance the two metrics so that they are harmonious with each other through Eq. 4. An evaluation called Area Under the Receiver Operating Characteristic Curve (ROC-AUC) is needed to find out how well the model can differentiate between positive and negative classes (ROC-AUC), which is shown in Eq. 5. In the context of an imbalance, the ROC AUC evaluation data sometimes provide misleading information so that an evaluation metric called Precision-Recall Curve (PRAUC) is used.

Results and Discussion
Four oversampling techniques are carried out depending on the formal diagram scheme being used: ROS, SMOTE, Borderline-SMOTE, and ADASYN. Three machine learning methods are employed to determine the best candidate to solve credit card fraud, namely Random Forest, Logistic Regression, and K-NN.
(1) Table 1 shows the classification accuracy, precision, recall, F1-score, ROC-AUC, and PRAUC of the four oversampling techniques on the credit card fraud dataset. In the use of raw data, the results are shown to be very good. However, this is the main problem because the data's composition is highly unbalanced, resulting in a bias in the classification model. The oversampling technique is needed to increase the number of minority classes in data imbalances [23].   The precision value determines how many relevant predictive values were correctly predicted. The best score is Borderline-SMOTE, with a result of 09474. Interestingly, when observing the model, Random Forest's results give the highest average precision compared to other models because Random Forest uses bagging approach in the decision tree process to provide lower variance [24].
The next metric is recall, which determines how many the actual predictive value of the data. ADASYN held the best score with a result of 0.9286. ADASYN has the advantage of calculating the density of data distribution so that the results of the duplication data provided can be focused on one particular area and strengthen the results of recall evaluation.
Precision and recall have mutually dependent characteristics, which are called trade-offs. Through this, the F1-score plays an important role in knowing the alignment between the precision and recall values by calculating the average value for each evaluation value from the precision and recall metrics. The best F1 score is Borderline-SMOTE, with a score of 0.9000. This high result indicates that Borderline-SMOTE provides the most balanced results compared to other oversampling values because the resulting data duplication is on the margins of the minority data to strengthen the differentiating value between the minority class and the majority class. Furthermore, the ROC-AUC value will determine how well the machine learning model distinguishes between negative and positive classes. Among all the oversampling techniques in this study, the highest result is obtained by ROS with a value of 0.9822. It can be seen in Fig. 4 that the combination of machine learning and oversampling techniques can distinguish negative and positive classes with satisfactory results with a value range of 0.93-0.98. Even though the ROC-AUC result is relatively high, there is still a possibility that the results presented could be misleading when using a highly unbalanced dataset. Therefore, PRAUC is used to find out the reliability of the resulting classification model. PR-AUC identifies how well the average precision is generated at each threshold result of the recall. The best result is ROS, with a score of 0.8603, beating all other competitors in the oversampling techniques. According to Fig. 5, it can be seen that the K-NN gives results between 0.58-0.82, which shows that this model is not reliable for credit card fraud cases. Furthermore, the Logistic Regression results show that the value between 0.74-0.76, giving an unreliable average result. In Table 1, the precision results generated by Logistic Regression are terrible. Finally, Random Forest yields a result value between 0.84-0.86, which shows that this model is reliable for credit card fraud cases.

Conclusion
The effect of high-class imbalance can affect the overall classification performance. The machine learning model will predict most of the data included in the majority class so that the evaluation of the evaluation results gives biased results. Results without using oversampling give high evaluation results indicating bias in the model. Bias in the model occurs because the prediction model for the minority class has no impact on the model so that an oversampling technique is needed to avoid bias in the results of the model evaluation. In this study, the oversampling technique can avoid bias in the overall evaluation results. The test results of the oversampling technique on the imbalance of credit card fraud data gave quite good results with an accuracy of above 0.89. However, the accuracy results sometimes give misleading results, so that the evaluation of other metrics are needed, such as precision, recall, F1score, ROC-AUC, and PRAUC. Borderline-SMOTE gave the best results with an average PRAUC of 0.8131. Borderline-SMOTE achieves the highest results because it duplicates the margins of the minority data so that it strengthens the difference between the minority and majority class data. ROS obtained the second position with an average PRAUC result of 0.7961. This result is obtained by duplicating based on minority data and randomly. Even though it has a high result, ROS's biggest drawback is that there is little variation in the duplicated data due to the duplication of data based on the original data. SMOTE gives an average PRAUC result of 0.7307. SMOTE duplicates data based on the closest neighbor to minority data so that the characteristics of SMOTE are sensitive to outliers, which can result in minority data being duplicated into the majority data area. Finally, ADASYN with an average PRAUC result of 0.7189. ADASYN works by duplicating data based on the density of the data distribution for the minority class. The more scattered the minority data, the higher possibility of resulting one sample data of minority class.
The best machine learning model is generated by Random Forest with an average accuracy value of 0.9996, precision of 0.9121, recall of 0.8541, F1-score of 0.8819, ROC-AUC of 0.9448, and PRAUC of 0.8513. The Random Forest has a characteristic of being more resistant to overfitting, usually when minority data are accidentally duplicated into the majority data area. Interestingly, Logistic Regression gives poor precision results with an average value of 0.0486. Poor precision results from Logistic Regression affects the resulting F1 score because there is a trade-off between precision and recall. The best combination of oversampling and machine learning method is Borderline-SMOTE with Random Forest with 0.9997 accuracy, 0.9474 precision, 0.8571 recall, 0.9000 F1 score, 0.9388 ROC-AUC, and 0.8581 PRAUC. Through the results of this test, it can be seen that the oversampling technique can provide actual results from data imbalances and can avoid bias in the entire machine learning model. Selecting the right oversampling technique is critical to do in order to maximize the evaluation results of the predictive model.

Declaration of Competing Interest
We declare that we have no conflict of interests.