The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> In machine learning problems, differences in prior class probabilities -- or class imbalances -- have been reported to hinder the performance of some standard classifiers, such as decision trees. This paper presents a systematic study aimed at answering three different questions. First, we attempt to understand the nature of the class imbalance problem by establishing a relationship between concept complexity, size of the training set and class imbalance level. Second, we discuss several basic re-sampling or cost-modifying methods previously proposed to deal with the class imbalance problem and compare their effectiveness. The results obtained by such methods on artificial domains are linked to results in real-world domains. Finally, we investigate the assumption that the class imbalance problem does not only affect decision tree systems but also affects other classification systems such as Neural Networks and Support Vector Machines. <s> BIB001 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Electricity consumer dishonesty is a problem faced by all power utilities. Finding efficient measurements for detecting fraudulent electricity consumption has been an active research area in recent years. This paper presents a new approach towards Non-Technical Loss (NTL) analysis for electric utilities using a novel intelligence-based technique, Support Vector Machine (SVM). The main motivation of this study is to assist Tenaga Nasional Berhad (TNB) in Malaysia to reduce its NTLs in the distribution sector due to electricity theft. The proposed model preselects suspected customers to be inspected onsite for fraud based on irregularities and abnormal consumption behavior. This approach provides a method of data mining and involves feature extraction from historical customer consumption data. The SVM based approach uses customer load profile information to expose abnormal behavior that is known to be highly correlated with NTL activities. The result yields classification classes that are used to shortlist potential fraud suspects for onsite inspection, based on significant behavior that emerges due to irregularities in consumption. Simulation results prove the proposed method is more effective compared to the current actions taken by TNB in order to reduce NTL activities. <s> BIB002 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Traditional classification algorithms can be limited in their performance on highly unbalanced data sets. A popular stream of work for countering the problem of class imbalance has been the application of a sundry of sampling strategies. In this paper, we focus on designing modifications to support vector machines (SVMs) to appropriately tackle the problem of class imbalance. We incorporate different ldquorebalancerdquo heuristics in SVM modeling, including cost-sensitive learning, and over- and undersampling. These SVM-based strategies are compared with various state-of-the-art approaches on a variety of data sets by using various metrics, including G-mean, area under the receiver operating characteristic curve, F-measure, and area under the precision/recall curve. We show that we are able to surpass or match the previously known best algorithms on each data set. In particular, of the four SVM variations considered in this paper, the novel granular SVMs-repetitive undersampling algorithm (GSVM-RU) is the best in terms of both effectiveness and efficiency. GSVM-RU is effective, as it can minimize the negative effect of information loss while maximizing the positive effect of data cleaning in the undersampling process. GSVM-RU is efficient by extracting much less support vectors and, hence, greatly speeding up SVM prediction. <s> BIB003 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> The volume of energy loss that Brazilian electric utilities have to deal with has been ever increasing. The electricity concessionaries are suffering significant and increasing loss in the last years, due to theft, measurement errors and many other kinds of irregularities. Therefore, there is a great concern from those companies to identify the profile of irregular customers, in order to reduce the volume of such losses. This paper presents the proposal of an intelligent system, composed of two neural networks ensembles, which intends to increase the level of accuracy in the identification of irregularities among low tension consumers. The data used to test the proposed system are from Light S.A. Company, the Rio de Janeiro concessionary. The results obtained presented a significant increase in the identification of irregular customers when compared to the current methodology employed by the company. <s> BIB004 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Nowadays the electric utilities have to handle problems with the non-technical losses caused by frauds and thefts committed by some of their consumers. In order to minimize this, some methodologies have been created to perform the detection of consumers that might be fraudsters. In this context, the use of classification techniques can improve the hit rate of the fraud detection and increase the financial income. This paper proposes the use of the knowledge-discovery in databases process based on artificial neural networks applied to the classifying process of consumers to be inspected. An experiment performed in a Brazilian electric power distribution company indicated an improvement of over 50% of the proposed approach if compared to the previous methods used by that company. <s> BIB005 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Abstract High values of total percentage losses of electricity in Serbia’s distribution network and a very small percentage of success in detecting electricity fraud indicate a need for a sophisticated means of detecting fraud perpetrators. In order for this task to be performed efficiently, the authors of the paper emphasize the need for a suitable and comprehensive use of the billing system or the database of the amounts of invoiced spent electricity in accounting periods and other relevant data regarding registered customers. For that purpose, they suggest using the rough set theory and give a general approach to its use. The point of the paper is forming a criterion for the estimation of accurate (suitable) discretization of original data. The criterion is based on the amount of lost not invoiced electricity due to electricity fraud. Based on consumption characteristics of detected fraud of customers whose measurement points were regularly (monthly) read, a list of the suspicious customers will be formed which will serve as the basis for sending expert teams to specified locations with the task to confirm or dismiss the fraud suspicion. <s> BIB006 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Electricity theft is a major concern for the utilities. With the advent of smart meters, the frequency of collecting household energy consumption data has increased, making it possible for advanced data analysis, which was not possible earlier. We have proposed a temperature dependent predictive model which uses smart meter data and data from distribution transformer to detect electricity theft in an area. The model was tested for varying amounts of power thefts and also for different types of circuit approximations. The results are encouraging and the model can be used for real world application. <s> BIB007 </s> The Challenge of Non-Technical Loss Detection using Artificial Intelligence: A Survey <s> A. Class imbalance and evaluation metric <s> Non-technical losses (NTL) such as electricity theft cause significant harm to our economies, as in some countries they may range up to 40% of the total electricity distributed. Detecting NTLs requires costly on-site inspections. Accurate prediction of NTLs for customers using machine learning is therefore crucial. To date, related research largely ignore that the two classes of regular and non-regular customers are highly imbalanced, that NTL proportions may change and mostly consider small data sets, often not allowing to deploy the results in production. In this paper, we present a comprehensive approach to assess three NTL detection models for different NTL proportions in large real world data sets of 100Ks of customers: Boolean rules, fuzzy logic and Support Vector Machine. This work has resulted in appreciable results that are about to be deployed in a leading industry solution. We believe that the considerations and observations made in this contribution are necessary for future smart meter research in order to report their effectiveness on imbalanced and large real world data sets. <s> BIB008
Imbalanced classes appear frequently in machine learning, however, this fact is mostly not addressed in the literature. This topic is well covered for example in BIB001 and BIB003 . The class imbalance also affects the choice of evaluation metrics. Most NTL detection research such as BIB005 ,  , BIB002 ,  and BIB006 also ignore this topic and report high accuracies or recalls. The following examples demonstrate why those performance measures are not suitable for NTL detection in imbalanced data sets: for a test set containing 1K customers of which 999 have regular use, (1) a classifier always predicting non-NTL has an accuracy of 99.9%, whereas in contrast, (2) a classifier always predicting NTL has a recall of 100%. While the classifier of the first example has a very high accuracy and intuitively seems to perform very well, it will never predict any NTL. In contrast, the classifier of the second example will find all NTL, but trigger many costly and unnecessary physical inspections. This topic is addressed for example in  and BIB004 , but do not use a proper single performance measure to describe the performance of a classifier performed on an imbalanced dataset. For NTL detection, the goal is to reduce the false positive rate (FPR) to decrease the number of costly inspections, while increasing the true positive rate (TPR) to find as many NTL occurrences as possible. BIB008 proposes to use a receiver operating characteristic (ROC) curve, which plots the TPR against the FPR. The area under the curve (AUC) is a performance measure between 0 and 1, where any binary classifier with an AUC > 0.5 performs better than random guessing. In order to assess a NTL prediction model using a single performance measure, the AUC was picked as the most suitable one. In the preliminary work of BIB008 , we noticed that the precision usually grows linearly with the NTL proportion in the data set. It is therefore not suitable for low NTL proportions. However, we did not notice this for the recall and made observations of non-linearity similar to the work of BIB007 summarized in Table I . With the limitation of precision and recall as isolated performance measures, the F 1 score did not prove to work as a reliable performance measure. We believe that it is necessary to investigate more into this topic in order to report reliable and imbalance-independent results that are valid for different levels of imbalance. The Matthews correlation coefficient (MCC) defined in  : measures the accuracy of binary classifiers taking into account the imbalance of both classes, ranging from −1 to +1. We believe that this measure should be assessed further for NTL detection.