Energy big data: A survey <s> B. DATA CORRELATION ANALYSIS <s> Preface. Notation. 1. Background. 2. Introduction to SIME. 3. Sensitivity Analysis. 4. Preventive Analysis and Control. 5. Integrated TSA&C Software. 6. Closed-Loop Emergency Control. 7. Retrospect and Prospect. Appendices. References. Index. <s> BIB001 </s> Energy big data: A survey <s> B. DATA CORRELATION ANALYSIS <s> This paper presents a novel data mining framework for the exploration and extraction of actionable knowledge from data generated by electricity meters. Although a rich source of information for energy consumption analysis, electricity meters produce a voluminous, fast-paced, transient stream of data that conventional approaches are unable to address entirely. In order to overcome these issues, it is important for a data mining framework to incorporate functionality for interim summarization and incremental analysis using intelligent techniques. The proposed Incremental Summarization and Pattern Characterization (ISPC) framework demonstrates this capability. Stream data is structured in a data warehouse based on key dimensions enabling rapid interim summarization. Independently, the IPCL algorithm incrementally characterizes patterns in stream data and correlates these across time. Eventually, characterized patterns are consolidated with interim summarization to facilitate an overall analysis and prediction of energy consumption trends. Results of experiments conducted using the actual data from electricity meters confirm applicability of the ISPC framework. <s> BIB002 </s> Energy big data: A survey <s> B. DATA CORRELATION ANALYSIS <s> This paper provides a synchrophasor data analysis methodology that leverages statistical correlation techniques in order to identify data inconsistencies, as well as power system contingencies. Included in this analysis are the techniques for data management that are used to process the high-granularity and high-cardinality data gathered from synchrophasor sensors - also know as Phasor Measurement Units (PMUs). This research utilizes real, archived PMU data obtained from the Western Electric Coordinating Council (WECC) in order to show that this methodology is not only feasible, but extremely useful for power systems monitoring, decision support, and planning purposes. The results presented indicate preliminary identification of PMU data issues, as well as power system instabilities. <s> BIB003 </s> Energy big data: A survey <s> B. DATA CORRELATION ANALYSIS <s> The large size and complexity of datasets in Big Data need specialized statistical tools for analysis and we use R for correlation analysis of our data set. This paper explores the correlation analysis through best fit linear regression of quantitative variables with help of the demonstration based on scatter plots and linear regression best fit line. The analysis demonstrated in this paper is scalable to Big Data in any other context where the quantitative variables are clearly delineated. R provides multiple techniques and inferences to statistical analysis of dataset, this paper however explores the correlation between quantitative variable establishing the extent of dependability between them using R functions. The correlation and best fit line functions of R i.e. Cor () and abline(lmout) respectively are significantly explored. <s> BIB004
The big data technologies in data analysis involve correlation analysis, data mining and machine learning. Through data analysis techniques, valuable information can be extracted from large amount of big data in a power system. Correlation is a well-known statistical and mathematical method for the compatibility analysis of large data sets. Meier et al. BIB003 successfully used Pearson Product-Moment correlation to determine how well data is linearly correlated. Given two independent input data sets of X , Y with the length of N , and X , Y being either the phase data values of two PMU sites or the momentary magnitudes, the Pearson correlation submits to a correlation coefficient C (C ∈ [−1, 1]), which is based on the following equation: Two application-specific improvements and modifications have been made to this mathematical formula. Firstly, because the algorithm was made incremental, each i data point (x i , y i ) could be gained from the Phasor Data Concentrator (PDC) engine and incorporated into its correlation coefficient at once. Thus, there is no need to directly calculate each average, summation, and standard deviation at each time step repeatedly. This helps reduce the operation time. Secondly, the correlation algorithm is set to maintain correlation information through varying windows of time. The stream data structure actually maintains multiple separate points to terminal positions of each defined sliding window. However, separate lists for each sliding window are not created. Rather, points from a single list are controlled to minimize memory usage and copies of data which are already managed. Correlation analysis techniques can figure out correlation or causal relationship and have been widely applied in power system for making investment decisions and forecasting the electric power load. As mentioned above, data mining ranges in the most popular techniques of data analysis, which is mainly applied for prediction and presentation in power grid. Machine learning is also widely applied in energy system. Pandy et al. BIB004 used correlation analysis of big data to support machine learning through best fit linear regression of large data sets. In addition, correlation analysis also applies for monitoring power equipment conditions, assessing transient stability, etc BIB001 . De Silva et al. BIB002 proposed an Incremental Summarization and Pattern Characterization (ISPC) framework using data mining technology to address problems in energy consumption analysis. This framework has two basic functions: incremental learning and interim summarization. Incremental learning can extend dynamic self-organization of senor nodes in power grid with the dynamic topology preservation technology. Using a structure-adapting feature map, this technology can ensure natural groupings in the data being exposed. The structure adaption is affected by a total quantization error E i,t , predefined values x and w, a spread factor which controls growth of the dynamic feature map: where H i means the ith hit node and D refers to the numbers of energy environment location phases. When E i,t exceeds the spread factor, the network structure will be adapted to fit this change, through growing new energy nodes or distributing the error to neighboring nodes. Interim summarization uses dimensional model representing the power environment in order to generate online digests of flow data. The digests facilitates data analysis in various directions, such as trend analysis, granularity exploration and ad-hoc querying.