DataIQ A Machine Learning Approach to Anomaly Detection for Energy Performance Data Quality and Reliability
The paper develops and tests a strategy for evaluating and improving the data quality and analysis of benchmarking data from privatelyand publicly owned buildings. We utilize machine learning tools to support the District of Columbia District Department of the Environment (DDOE) in improving the quality of building energy data collected through its mandatory building rating and disclosure policy. Using energy disclosure data from calendar year 2013, this paper presents a new data quality rating algorithm and grading system known as the Data Integrity and Quality (DataIQ) score to rank the relative reliability of reported data and provide a tool to improve the identification and prediction of data quality concerns going forward. We apply non parametric statistical anomaly detection techniques to identify data quality and reliability concerns in self reported building energy benchmarking data. This approach creates a foundation for more robust and precise analysis of city energy data that can provide policymakers with additional insight to improve the reliability of building benchmarking data analysis and inform the design of data driven energy efficiency policies. INTRODUCTION The District of Columbia and the City of New York were the first two jurisdictions to adopt mandatory benchmarking and disclosure laws (City of New York 2012, Hsu 2014 Kontokosta 2012, 2013) These laws require large buildings (in both cities, buildings over 50,000 gross square feet) to annually benchmark their energy and water performance using the U.S. Environmental Protection Agency's Energy Star Portfolio Manager software, and report the results to the cities, which make the data available online. Mandatory benchmarking and Disclosure programs have three fundamental purposes: (1) to provide building owners managers with better information about the efficiency of their own properties, and how those properties compare to local and national peers; (2) to drive market transformation by allowing market actors to easily compare the performance of properties when leasing, buying, or investing; and (3) to provide policy makers and program administrators with better information for planning, program design, and targeting. (Hart 2015; Keicher et al. 2012) All of these uses depend fundamentally on the reliability of the reported data (Hsu 2014; Kontokosta 2013; 2015; Palmer and Walls 2015) Building owners, managers, and other market actors must have confidence in the reliability of the benchmarking data, and subsequent analysis, in order to make decisions that save money and energy. And cities, utilities, and contractors must have confidence in the reliability of the data to design policies and programs that address market needs and drive change. If the market comes to question the quality of benchmarking data, the resulting uncertainty will undermine the potential for data driven market transformation and a precipitate a loss of trust among decision makers that could be difficult to regain. Thus, both actual or objective and perceived or contextual data reliability are needed to move from the superficial availability of information to proactive decisions based directly on insights derived 12 1 (c)2016 ACEEE Summer Study on Energy Efficiency in Buildings from it (Wang and Strong 1996) For this reason, improving the completeness and quality of the benchmarking data is a very high priority of many benchmarking programs, including the District's. In the implementation of the benchmarking law, the District of Columbia Department of Energy and Environment (DOEE) which implements the benchmarking law in Washington, DC, has found common self reported data errors. In general, we can classify these errors with respect to the completeness of the data, its consistency, and compliance rate (Pipino, Lee, and Wang 2002) Building on the benchmarking data quality issues in Kontokosta (2013) the types of data errors specific to building energy disclosure data can be divided into five categories: 1. Fatal Errors: User error that leads to a report being submitted without any metrics, such as Gross Floor Area, Energy Use Intensity (EUI) Water Use Intensity (WUI) These errors are usually caused by the respondent not entering in complete meter data for the whole year or setting the "active date" for meter or space use values incorrectly; 2. Energy Data: Inaccurate or incomplete energy consumption information due to either data collection errors (not reporting all the energy meters for the building) data entry error (e.g. incorrect units) or utility company errors (incorrect billing or aggregation) 3. Floor Area: Inaccurate square footage based on either the use of unverified tax data square footage, which can often be wrong, or incorrect understanding of what spaces in the building need to be included (e.g. net vs. gross square footage) 4. Space Use: Inaccurate space use attributes due to data entry errors, confusion about the requirements, or the use of default values; and 5. ENERGY STAR Scoring Issues: Problems in the methodology of ENERGY STAR Portfolio Manager itself (e.g. changes in site to source energy ratios, or use of outdated or insufficient reference data sets) Fatal errors can be minimized when cities work directly with building owners to correct these (relatively) simple user errors. In DC, 25% of initial disclosure reports contained this type of error; however, the District has reduced this rate to just 3% through compliance assistance and enforcement. Starting in late 2015, DC is now enforcing compliance standards for observed fatal errors, considering them to be equivalent to non submittal. The fifth error category problems with Energy Star Portfolio Manager is an important area for academic investigation, but beyond the scope of this paper (for additional information, see Kontokosta 2015 and Hsu 2014) Moreover, the value for city governments of using a federally supported, industry standard tool that is common across jurisdictions is an important consideration when evaluating the lack of local control and methodological concerns about the Portfolio Manager software itself. A fundamental concern of cities trying to implement benchmarking laws are errors relating to EUI and space use values (which drive the Energy Star score) Some jurisdictions, such as Chicago, Illinois and Montgomery County, Maryland, have started to require third party verification of data quality; in other jurisdictions, the burden to ensure data quality falls more heavily on the city. DOEE does operate a Benchmarking Help Center that fields between 1,500 and 2,000 requests for assistance a year. However, the scale of the data makes a purely manual approach to data quality verification insufficient DC has approximately 1,600 benchmarking reports, while a larger city like NYC receives over 14,000 each year. Therefore, a methodology to triage reports and flag those with data quality concerns is needed. 12 2 (c)2016 ACEEE Summer Study on Energy Efficiency in Buildings The paper develops and tests a transparent and robust strategy for evaluating and improving the data quality and analysis of benchmarking data from privatelyand publiclyowned buildings. We utilize machine learning tools to support jurisdictions like the District of Columbia in improving the quality of building energy data collected through its mandatory building rating and disclosure policy. Using energy disclosure data from calendar year 2013, this paper presents a new data quality rating algorithm and grading system known as the Data Integrity and Quality (DataIQ) score to rank the relative reliability of reported data for the largest building sectors, and provide a tool to improve the identification and prediction of data quality concerns going forward. We apply non parametric statistical anomaly detection techniques to identify data quality and reliability concerns in self reported building energy benchmarking data, focusing here on office buildings. The next section describes the data and data cleaning methodology, followed by a discussion of the DataIQ methodology and results. We conclude with a discussion of the implications of the model and its potential application as part of city energy disclosure policies. DATA AND DESCRIPTIVE STATISTICS The data for this analysis include all properties subject to the requirements of the Clean and Affordable Energy Act of 2008 that reported data to the Washington, DC Department of Energy and Environment for calendar year 2013. The analyzed dataset consists of all private and public buildings subject to the energy disclosure requirement that provided data through September 4, 2015. After the removal of duplicate entries, the 2013 dataset includes a total of 1,774 unique records, divided between private buildings (1,456) public (government) buildings (274) and public housing (44) Due to non trivial errors and omissions of various reported fields specific to multi family housing, we discuss only office building data here. Multi stage data cleaning represents a critical component of building energy analytics and the ability to extract reliable insight from energy data. First, the datasets are merged to create an integrated dataset of private and public buildings for each respective year. As part of this process, variable headers and other entries were cleaned of atypical characters and formatting inconsistencies. The combined dataset was then joined with the provided tax data based on Property ID and Square Suffix Lot (SSL) information. This process successfully matched disclosure and tax lot data for 1,169 properties in the 2013 dataset. Next, entries with duplicate Property IDs were removed, with only the most recent entry retained. Where feasible and appropriate, missing values in the most recent Property ID record were imputed from earlier submissions for the same Property ID. The resulting dataset contained 1,774 for the 2013. Of the reported fields,