Online cleaning method of power grid energy anomaly data based on improved random forest

Aiming at the problem of high root mean square error of traditional power grid energy anomaly data online cleaning, a power grid energy anomaly data online cleaning method based on improved random forest is designed. Firstly, an outlier data recognition model of isolated forest is designed to identify outliers in the data. Secondly, an improved random forest regression model is established to improve the adaptability of random forest to mixed abnormal data, and the data trend is fitted and predicted. Finally, the improved random forest data cleaning method is used to compensate the missing data after removing the mixed abnormal data, so as to clean the abnormal energy data of the power grid. The experimental results show that when the amount of power grid energy anomaly data increases, the cleaning root mean square error of the experimental group is significantly lower than that of the control group. The method in this paper solves the problem of high root-mean-square error in the online cleaning of abnormal data of traditional grid energy.


Introduction
The power grid contains huge real-time data, which is closely related to the safe operation of the power system and plays a decisive role in the safety and reliability of the power grid. The power grid needs to generate and consume a lot of electric energy in the process of working. Under the condition of normal work, it will still produce bad data, affect the decision-making of the system and threaten the safety of the whole power grid. Data cleaning is not only applied in power grid, but also in the fields of data mining and total data quality management. The definitions of data cleaning are different in different fields, and there is no unified definition of data cleaning. Therefore, using professional technology to detect and correct bad data will help to ensure the normal operation of power grid and the normal supply of power energy. So far, scholars at home and abroad have done a lot of theoretical and practical research on the detection and correction of power grid energy big data, and summarized two widely used cleaning methods. One is the identification and cleaning method of abnormal data of power grid operation based on Spark framework. Abnormal data are extracted from big data through data mining algorithms such as clustering or neural network, and cleaned under Spark framework. The other is the missing value cleaning method of power grid data based on improved low rank matrix completion. This method mainly sets the threshold according to a certain confidence level, carries out hypothesis test combined with the relevant knowledge of probability theory, and then completes the missing value of data by improving the low rank matrix to realize anomaly cleaning. In the actual operation of the power grid, it is very difficult to ensure the accuracy and completeness of user electricity data collection. Affected by sensor failures, transmission line failures, natural weather and other complex factors, users' electricity consumption data will have varying degrees of data omissions and errors. This article refers to these user electricity consumption data as abnormal data. In the above two methods, the data recognition is abnormal due to too many repeated searches, or the normal data is judged as bad data, resulting in a high root mean square error in practical applications.
The data set does not need standardization, and the training speed is fast. It can be applied to largescale data sets. Default values can be processed (as a separate class) without additional processing. Due to the out of bag data (OOB), the unbiased estimation of the real error can be obtained in the process of model generation without losing the amount of training data. In the training process, the interaction between features can be detected, and the importance of features can be obtained, which has a certain reference significance. Because each tree can be generated independently and simultaneously, it is easy to make a parallelization method. Because of its simple implementation, high precision and strong anti overfitting ability, it is suitable to be used as a benchmark model in the face of nonlinear data. Based on this, this paper designs an online cleaning method of power grid energy anomaly data based on improved random forest to solve the problems existing in the previous cleaning methods.

Improved random forest
In order to adapt to the compensation of mixed missing data, the random forest algorithm is improved to obtain an improved random forest algorithm with strong adaptability [1][2] . The X excluding abnormal data is linearly interpolated, and then the filling matrix containing target compensation variables is obtained through matrix transformation. Then, the random forest regression model is used to predict the filling matrix, and the integration idea is used to output tb times [3][4] . The specific methods for improving random forest are as follows: In step 1, the missing value in X is interpolated by linear interpolation method to obtain matrix R, which is expressed as: In formula (1): the sample size of R and X is the same, which is m × n dimensional matrix, n is the number of variables, and m is the number of data samples contained in a single variable.
In step 2, take column i in X as the target filling column, and the remaining n-1 column in R except column i as the relevant variable column to form the filling matrix Rfill, I = 1, 2, 3,..., n, expressed as: In step 3, a training matrix Dzv is randomly selected from D to complete the establishment of random forest regression model.
In step 4, take the relevant variable column in Rfill as the input and the target filling column as the output to predict the missing value of column i to obtain the t-th prediction value y(t), t = 1, 2, 3,..., tb.
Step 5 if t < tb, then t = t + 1, return to step 4; otherwise, take the average value of tb predictions as the final compensation value yz, and replace the missing value in xi with yz to complete the update of matrix X.
Step 6 if i < n, i = i + 1, repeat the above steps; otherwise, stop prediction and complete data compensation for X [5][6] . The above is to improve the overall step flow of random forest algorithm.

Classification of grid energy anomaly data
According to the analysis of the original data of power grid energy, the abnormal data of power grid energy are mainly divided into the following three categories: outlier data, duplicate data and missing data. Outliers are data that deviate significantly from the rest of the values [7] . Duplicate data is the data with the same value at multiple times. According to the dynamic characteristics of power grid operation process, this kind of data is determined as abnormal data. Missing data refers to data not collected at a certain time or data vacancy [8] . However, there are many energy data variables and parameters in the power grid, and the abnormal data do not appear in a single type, but often appear in a variety of abnormal characteristics at the same time. At present, the research on online data cleaning of power grid energy anomalies is limited to cleaning methods for single type of abnormal data, and there is no mixed abnormal data cleaning method for different types. When the traditional probability distribution, clustering algorithm and intelligent algorithm are used to clean different types of mixed abnormal data, the effect is still difficult to meet the requirements of data quality during power grid operation. Therefore, this paper proposes a data cleaning method based on improved random forest to clean different types of mixed abnormal data.

Identification of grid energy anomaly data based on improved random forest
In order to quickly and accurately identify abnormal data, an abnormal data identification model based on isolated forest is constructed. IF is a decision tree ensemble learning method. IF algorithm first divides the known and continuous time data set randomly, and then realizes fast and accurate identification by using the difference between abnormal data and normal data. The IF model consists of a isolation tree, and the algorithm steps are as follows: Step 1 randomly select a sub sample matrix Xz from the training data matrix X as the set of root nodes of the b-th tree, where b = 1, 2, 3,..., a and a are the number of root nodes, and the training data matrix X and sub sample matrix Xz are expressed by the formula: In formula (4,5), the matrix is m × n dimensional matrix, n is the number of variables, and m is the number of data samples contained in a single variable; u is the number of samples, u = mn; Xz is mz × nz dimensional matrix, nz is the number of variables, mz is the number of data samples contained in a single variable, 0 < mz ≤ m, 0 < nz ≤ n.
Step 2 perform binary segmentation on Xz: randomly select a column vector xj, j∈{1, 2, 3,..., mz} from Xz, and randomly select a cutting point T from the set of column vectors xj, as shown in formula (6). (6), r is a random number between 0 and 1. If X z (i, j) ＜ T, all variables in row I of matrix X z are divided into left subtree nodes. If X z (i, j) ≥ T, all variables in row i of matrix X z are divided into right subtree nodes, i = 1, 2, 3,..., m z . When all the data in X z is divided, the left subtree node set forms matrix X left and the right subtree node set forms matrix X right .
Step 3 record the path length hn of the node where Xleft and Xright are located. hn is the number of edges from the root node to the current node. If hn is greater than or equal to the tree height hmax or the number of sets in nodes is less than or equal to mz, stop training and complete the construction of a single isolation tree; otherwise, perform binary division for Xleft and Xright, and repeat step 3. The tree height hmax is: hmax Ibu  Step 4 if b < a, b = b + 1, repeat step 2 and step 3, otherwise stop training and complete the establishment of if model. IF divides and isolates the data in X. the normal data needs to be divided and isolated for many times and is in a high-density area; abnormal data needs to be isolated by a few divisions and located in low-density areas.

Eliminate abnormal data of power grid energy
After the data is calculated by the IF model, different high and low density areas are formed. The density area where the data is located is reflected by calculating the abnormal value score of the data, and the data with high score is eliminated. The path length of xij output from IF model is hij, and xij is the element in matrix X. calculate the outlier score of xij through hij, and the formula is: In formulas (8 and 9), c(u) is the average path length of all data in X; ξ Is Euler constant; E(hij) is the average path length of data xij in a isolation tree. When the value of S(hij, u) approaches 0.5, it indicates that the data has no obvious abnormal state; When the value of S(hij, u) approaches 1, it indicates that the data is an abnormal value. After obtaining the outlier score of each data, combined with the equivalence and missing characteristics of the data, the following three kinds of data are eliminated from X: For the first data, xij with outlier score S(hij, u) greater than 0.75 in X is eliminated and replaced by 0. For the second data, replace the missing data in X with 0. For the third data, c1 or more consecutive xij with the same value in X are eliminated and replaced with 0.

Complete the online cleaning of abnormal energy data of power grid
From the above analysis, it can be seen that the identification and cleaning of non clean data is difficult to ensure the complete elimination of non clean data, especially the multi-dimensional nature of power grid energy abnormal data, which brings difficulties to cleaning. With the gradual improvement of the fineness of the cleaning, the operation performance of the knowledge base system will gradually decrease, and the information contained in the abnormal energy data of the power grid will be gradually lost, which can not quickly and effectively solve the problems caused by the non clean data. Data cleaning across data sources is mainly to ensure the correlation between metadata by matching the semantic constraints between multiple data sources. Assuming that the metadata feature set is sdfrewa, the following two constraints need to be met to complete metadata cleaning. First, if the two vertices of the edge representing the metadata relationship must be metadata objects, they are expressed by a formula, as shown in formula (10).
, , e f t E f t V    (10) In formula (10), e refers to the directed edge of the metadata feature;   , f t refers to the coordinate point corresponding to the metadata feature; E refers to the directed edge combination of metadata features; V refers to the vertex set of metadata features. Second, if the relationship between two metadata objects can only be one-way, it can be expressed by formula, as shown in formula (11).
On the basis of satisfying formula (11), the syntax and semantics of energy metadata characteristics in the power grid operation process can be analyzed, and then the energy effective information can be extracted and tracked, so as to complete the online cleaning of power grid energy abnormal data. Therefore, through data cleaning, the quality of power grid energy data can be improved.

Experimental preparation
The grid energy data information of a cross-border power enterprise in recent 5 years is selected and introduced into the simulation experiment software to build a virtual power database. Two computers with Windows 10 and 16GB memory capacity are selected to simulate the sending and receiving of power grid energy data. Another computer with Windows 10 and 32GB memory capacity is selected to simulate the transmission environment of power grid energy data, and the experiment is carried out with the abnormal power grid energy data as the sample data. This paper uses the online cleaning method of power grid energy abnormal data based on improved random forest design to clean the ICPEPT 2021 Journal of Physics: Conference Series 2108 (2021) 012067 IOP Publishing doi:10.1088/1742-6596/2108/1/012067 5 power grid energy abnormal data online, and then uses the traditional method to clean the power grid energy abnormal data online. 1000 grid energy anomaly data are randomly selected, of which 500 grid energy anomaly data are cleaned online by using the design method in this paper and set as the experimental group, and the other 500 grid energy anomaly data are cleaned online by using the traditional method and set as the control group. The root mean square error of the experimental group and the control group in the online cleaning of power grid energy anomaly data was compared. The lower the root mean square error, the higher the accuracy of the online cleaning method of power grid energy abnormal data, so as to obtain a higher accuracy online cleaning method of power grid energy abnormal data.

Experimental results and analysis
According to the above experiment preparation, complete the comparative experiment, record the data information generated during the experiment, and display the experimental results in the form of tables. The specific experimental results are shown in Table 1.  Table 1, when the amount of grid energy abnormal data increases, the cleaning root mean square error of the experimental group is significantly lower than that of the control group, which can realize high-precision online cleaning of grid energy abnormal data. The main reason is that the experimental group can realize intermittent deletion mixed continuous deletion data cleaning based on the improved random forest. Therefore, the comparative experiments show that the online cleaning method proposed in this paper can effectively reduce the root mean square error of online cleaning of power grid energy anomaly data and improve the online cleaning accuracy. The mean square error of the cleaning data using this method is less than 0.21, while the mean square error of the cleaning data of the control group is higher than 0.75. It further ensures the quality of power grid energy data, meets the needs of power grid construction at the present stage, and has higher application value.

Conclusion
Aiming at the problem of energy abnormal data in the process of power grid operation, an online cleaning method of power grid energy abnormal data based on improved random forest is proposed in this paper. Firstly, the random forest algorithm is used to identify the outlier data. Then, outlier data, duplicate data and missing data are eliminated to obtain the mixed type missing data set. Finally, the improved random forest algorithm is used to compensate the missing data sets of mixed types. The effectiveness of IRF data cleaning algorithm is verified by taking the actual sewage data, and the following conclusions are obtained: the random forest algorithm can eliminate the outlier data in the power grid energy data. In the mixed type missing data set, the improved random forest algorithm has better cleaning effect than other algorithms, and is suitable for cleaning mixed type abnormal data. For data cleaning algorithms, some research institutions have proposed data preprocessing, sorting neighbor method, multiple traversal data cleaning method, cleaning with domain knowledge, integrated data cleaning with database management system and so on. Based on the differences between Chinese data and Western data, Chinese data cleaning not only transplants the cleaning methods of Western data, but also has its own unique cleaning methods. According to the deficiency of current data cleaning research, the main research directions of data cleaning in the future are: the research and development of Chinese data cleaning tools; do in-depth research on the application of data mining methods in the field of data cleaning; the efficiency of duplicate record identification needs to be further improved; cleaning of unstructured data; interoperability between data cleaning tools; universality of data cleaning scheme. The improvement and optimization of Chinese data cleaning technology will be further studied in the follow-up.