Physics
Computer Science-2021.gz:143153
Computer Science
dca43be3478726d50af866dc0b7449b9b1a01d1b
Evaluation performance recall and F2 score of credit card fraud detection unbalanced dataset using SMOTE oversampling technique

Unbalanced data becomes an interesting research and continues to be studied because of its uniqueness. Unbalanced data requires special treatment prior to making the data balance. In this paper, our study to investigate the performance of unbalanced dataset using diverse oversampling proportion. We use SMOTE to gerentae new syntethic data, then we classify using random forest algorithm. In our experiment we generate new sampling with start 20%, 40%, 60%, 80%, and 100% of majority class, so that the data balancing until 50%: 50%. Each new generated data, we train the data using classification technique. Then, evaluate each algorithm performance. We show that the highest F2 score i.e: 85.34 and 84.93. The new data generated is 60% of majority class, result F2 score 85.34, then the new data generated from 100% of majority class result F2 score 84.93.


Introduction
Data mining has been widely used in various fields, ranging from education [1,2], health [3,4], marketing [5], to the economy [6]. Several studies in data mining, including examining how to improve the accuracy of the algorithm, for example in the field of health have been done [7,8]. Meanwhile, implementation in the economy sector, for example [9]. Research studies in the field of economics are quite interesting because, currently, a lot of transaction data or transaction activities are carried out online [10]. Online transactions via e-commerce make it easier in terms of transaction relations between countries. Online transaction activities have broken through time and space boundaries by utilizing internet technology, one of which is increasing export-import activities in China [11,12]. Payment for online transactions is made in non-cash, one of which is a credit card. However, the use of a credit card has risks, for example, when we found a fraud transaction.
Credit card fraud detection has become an interesting study, especially with the use of data mining. Dornadula has conducted a study on the use of machine learning for credit card fraud detection [13,14,15]. The implementation of this problem using on azure ML has been carried out by Shivanna [16]. The credit fraud dataset is the object of much study because of its uniqueness, namely the unbalanced nature of the data. Previously, Pozzolo has conducted research to conduct research on Calibrating Probability on Undersampling for Unbalanced [17]. Unbalanced data is a condition where the number of classes is not balanced. Data imbalances can affect data quality. Data quality can affect  [18].
The solution to overcome unbalanced data can be done by undersampling or oversampling. In this paper, we focus on the problem of oversampling, one of which is the Synthetic Minority Over-Sampling Technique (SMOTE). SMOTE simply creates synthetic data based on k nearest neighbors so as to create a balanced class distribution between the majority and minority classes [19]. SMOTE is an oversampling technique that is used to solve the problem of unbalanced datasets by modifying unbalanced datasets and producing a balanced dataset from unbalanced datasets. SMOTE distributes majority and minority class instances evenly thereby increasing the predictive accuracy of the minority class by creating synthetic instances of the minority class without causing overfitting due to synthetic sampling technique [20]. Research for banlancing data on SMOTE includes [21]. In this paper, we conduct a study to determine the classification performance using various class compositions using SMOTE. The data used refers to the credit card fraud detection dataset [17], where the data has an extreme unbalanced class

Methods
In this study, we performed data preparation: data collecting, data preprocessing. The dataset we use is credit fraud detection [17]. The research data contains transactions made with credit cards in September 2013 by European card holders. This dataset provides transactions that took place in two days. This data contains 492 frauds out of 284,807 transactions, its mean this dataset is extremely unbalanced because the positive class (fraud) has only 0.172% of all transactions. The composition of the class, not fraud compared to fraud was 99.828% : 0.172%. The dataset only contains numeric values, which are the result of PCA transformation, contains 28 attributes, and 1 Class is shown in Table 1. The next step is to perform data preprocessing, which is to normalize the data on the "Amount" attribute using the standard scaller (range -1 to 1), while the other attributes are not. Attribute "Time" is not normalized to see transaction times. Meanwhile, attribute V1 to V8 is the result of PCA transformation. Next, we did the experiment using the SMOTE [18] oversampling technique. Next, we conducted experiments with a diverse number of oversampling classes. From low oversampling to balanced oversampling. For each new data from the oversampling results, we conduct training using a classification algorithm. The classification algorithm used is logistic regression. Next, we perform algorithmic performance calculations. The performance in focus is the F2 score and Recall. We chose the F2 score according to the characteristics of the unbalanced data. We use Recall because of the characteristics of the dataset for fraud detection.

Results and Discussion
In our experiment dataset [17] show that the credit fraud dataset is extremely unbalanced. We did a visualization to show the proportion of fraud notes compared to fraud shown in Figure 2. In Figure 2 there are two bar charts, the left is Class "not fraud" (284,315 instances), while the right bar chart is "fraud" marked with a red circle (492 instances), with a comparison percentage of 99.828%: 0.172%.  (1) We then call the not fraud class the majority class, while the fraud class hereinafter will be called the minority class. In our experiment, we implement diverse oversampling proportional. Then we do classification and matrices the perform. We implement classification in python using sklearn module [23]. In our experiment, we resampled the minority class as x% of majority class as shown in Figure 3.   In our experiment, we can show that the recall and f2 performance increase along with the number of classes that are balanced. Figure 4 Provides the graphic that show the performance start from original data until balanced data (50:50). We can show that the recall value is still consistent in the last 3 experiments, amounting to 84.5%; 83.33%; 84.52%. The highest F2 score was in the proportion of data 62:38 amounting to 85.34%, while in the balanced class (50:50), it actually decreased to 84.93%.

Conclusion
We have make study to oversampling the extremely unbalanced dataset. We have done the experiment with a diverse proportion of oversampling technique using SMOTE. Then we classified using random forrest classification. In our experiment show that the highest F2 score and Recall in composition 62%: 38%. This oversampling the minority class is to the 80% of the majority class, with a recall value of 84.52 and F2 score of 85.34. Meanwhile, the proportion of 50% : 50% (oversampling as much as 100% of the majority class) was actually lower than 62%:38%, especially in the F2 score class, namely recall of 84.52 and F2 score of 84.93.