<jupyter_start><jupyter_text>Produce Eastern Demand Profile Using EIA DataIn the notebooks loacate in the **spp** and **miso** directories, we produced the following files:* a file for the hourly load in each sub-area. One profile for SPP which has 16 *legacy BAs* and one profile for MISO which has 6 *Local Resource Zones*.* a file listing each load bus in our model to a sub-area. One for SPP and one for MISO.<jupyter_code>import getpass
import os
import pandas as pd
from powersimdata.input.grid import Grid

from prereise.gather.demanddata.eia.clean_data import fix_dataframe_outliers, replace_with_shifted_demand
from prereise.gather.demanddata.eia.get_eia_data import get_ba_demand
from prereise.gather.demanddata.eia.map_ba import aggregate_ba_demand, get_demand_in_loadzone<jupyter_output><empty_output><jupyter_text>1. Download DataDemand data from EIA is imported using an API Key. Note that data for some BAs will not be found and this will be reported.<jupyter_code>eastern_ba = [
    'SWPP',
    'MISO',
    'AECI',
    'SPA',
    'EEI',
    'LGEE',
    'OVEC',
    'PJM',
    'NYIS',
    'ISNE',
    'TVA',
    'CPLW',
    'DUK',
    'YAD',
    'CPLE',
    'AEC',
    'SOCO',
    'SEPA',
    'SC',
    'TAL',
    'SCEG',
    'GVL',
    'JEA',
    'SEC',
    'FPC',
    'NSB',
    'TEC',
    'FMPP',
    'FPL',
    'HST']
start = pd.to_datetime('2016-01-01 00:00:00')
end = pd.to_datetime('2016-12-31 23:00:00')

key = getpass.getpass(prompt='api_key=')
eastern_demand = get_ba_demand(eastern_ba, start, end, key)
eastern_demand_2016 = eastern_demand.loc[start:end]
eastern_demand_2016<jupyter_output>api_key=········
Downloading EBA.SWPP-ALL.D.H
Downloading EBA.MISO-ALL.D.H
Downloading EBA.AECI-ALL.D.H
Downloading EBA.SPA-ALL.D.H
Downloading EBA.EEI-ALL.D.H
ERROR: EBA.EEI-ALL.D.H was found but has no data
Downloading EBA.LGEE-ALL.D.H
Downloading EBA.OVEC-ALL.D.H
Downloading EBA.PJM-ALL.D.H
Downloading EBA.NYIS-ALL.D.H
Downloading EBA.ISNE-ALL.D.H
Downloading EBA.TVA-ALL.D.H
Downloading EBA.CPLW-ALL.D.H
Downloading EBA.DUK-ALL.D.H
Downloading EBA.YAD-ALL.D.H
ERROR: EBA.YAD-ALL.D.H not found. invalid series_id. For key registration, documentation, and examples see https://www.eia.gov/developer/
Downloading EBA.CPLE-ALL.D.H
Downloading EBA.AEC-ALL.D.H
Downloading EBA.SOCO-ALL.D.H
Downloading EBA.SEPA-ALL.D.H
ERROR: EBA.SEPA-ALL.D.H not found. invalid series_id. For key registration, documentation, and examples see https://www.eia.gov/developer/
Downloading EBA.SC-ALL.D.H
Downloading EBA.TAL-ALL.D.H
Downloading EBA.SCEG-ALL.D.H
Downloading EBA.GVL-ALL.D.H
Downloading EBA.JEA-ALL.D.H
Do[...]<jupyter_text>2. Impute Missing ValuesFill missing data before decomposing into subareas.Use adjacent demand data to fill missing values using a series of rules:1. Monday: look forward one day2. Tues - Thurs: average of look forward one day and look back one day3. Fri: look back one day4. Sat: look forward one day5. Sun: look back one dayIf data is still missing after applying the above rules, week ahead and week behind data is used1. Monday: look forward two days 2. Tues: look forward two days3. Wed: average of look forward two days and look back two days4. Thurs: look back two days5. Fri: look back two days6. Sat - Sun: average of look back one week and look forward one weekIf data is still missing after applying the above rules, week ahead and week behind data is used:1. Mon - Sun: average of look back one week and look forward one week<jupyter_code>eastern_demand_2016 = replace_with_shifted_demand(eastern_demand_2016, start, end)
miso_subarea_demand = pd.read_csv(os.path.join('miso', 'miso_subarea_demand_profile.csv'), index_col=0)
spp_subarea_demand = pd.read_csv(os.path.join('spp', 'spp_subarea_demand_profile.csv'), index_col=0)
miso_subarea_demand.index = eastern_demand_2016.index
spp_subarea_demand.index = eastern_demand_2016.index<jupyter_output><empty_output><jupyter_text>3. Normalization of SPP and MISO ProfilesWe found the summation of subareas hourly profile does not match the hourly load profile from EIA for both SPP and MISO. We followed the EIA’s total but keep the hourly shape of each subarea. To do this, the load value for each subarea is scaled hourly proportionally to the relative ratio whereas the total matches the EIA report.<jupyter_code>(eastern_demand_2016['MISO'] - miso_subarea_demand.sum(axis=1)).plot(figsize = (10,8), grid=True)
(eastern_demand_2016['SWPP'] - spp_subarea_demand.sum(axis=1)).plot(figsize = (10,8), grid=True)<jupyter_output><empty_output><jupyter_text>Follow the profile shape of subarea profile and scale up to match the total absolute value from EIA for each hour.<jupyter_code>frac_subarea_miso = miso_subarea_demand.copy()
for zone in miso_subarea_demand.columns:
    frac_subarea_miso[zone] = miso_subarea_demand[zone].divide(
        miso_subarea_demand.sum(axis=1)).multiply(
        eastern_demand_2016['MISO'])
frac_subarea_spp = spp_subarea_demand.copy()
for zone in spp_subarea_demand.columns:
    frac_subarea_spp[zone] = spp_subarea_demand[zone].divide(
        spp_subarea_demand.sum(axis=1)).multiply(
        eastern_demand_2016['SWPP'])
(eastern_demand_2016['SWPP'].sum() - spp_subarea_demand.sum().sum())/spp_subarea_demand.sum().sum()
(eastern_demand_2016['MISO'].sum() - miso_subarea_demand.sum().sum())/miso_subarea_demand.sum().sum()
pd.concat([eastern_demand_2016['SWPP'].iloc[:72], 
           spp_subarea_demand.sum(axis=1)[:72]],axis=1).plot(figsize=(10,8), grid=True)
pd.concat([eastern_demand_2016['MISO'].iloc[:72],
           miso_subarea_demand.sum(axis=1)[:72]],axis=1).plot(figsize=(10,8), grid=True)
eastern_demand_2016['MISO'].plot(figsize=(10,8), grid=True)
frac_subarea_miso.plot(figsize=(10,8), grid=True)
miso_subarea_demand.plot(figsize=(10,8), grid=True)
eastern_demand_2016['SWPP'].plot(figsize=(10,8), grid=True)
frac_subarea_spp.plot(figsize=(10,8), grid=True)
spp_subarea_demand.plot(figsize=(10,8), grid=True)
eastern_demand_2016 = pd.concat([eastern_demand_2016, frac_subarea_miso, frac_subarea_spp], axis=1)<jupyter_output><empty_output><jupyter_text>4. Aggregate BAsSome BAs are very small. In these cases, BAs are agregated into BA areas as follows:<jupyter_code>ba_area ={
    'ISNE': ['ISNE'],
    'NYIS': ['NYIS'],
    'PJM': ['PJM', 'OVEC'],
    'AEC': ['AEC'],
    'SOCO': ['SOCO'],
    'TVA_LGEE': ['TVA', 'LGEE'],
    'Florida': ['TAL', 'JEA', 'GVL', 'SEC', 'FPC', 'NSB', 'FMPP', 'TEC', 'FPL', 'HST'],
    'AECI': ['AECI'],
    #'MISO':['MISO','AECI'],
    'LRZ1': ['LRZ1'],
    'LRZ2_7': ['LRZ2_7'],
    'LRZ3_5': ['LRZ3_5'],
    'LRZ4': ['LRZ4'],
    'LRZ6': ['LRZ6'],
    'LRZ8_9_10': ['LRZ8_9_10'], 
    #'SWPP':['SWPP','SPA'],
    'CSWS': ['CSWS', 'SPA'],
    'EDE': ['EDE'],
    'GRDA': ['GRDA'],
    'KACY': ['KACY'],
    'KCPL_INDN': ['KCPL_INDN'],
    'LES': ['LES'],
    'MPS': ['MPS'],
    'NPPD': ['NPPD'],
    'OKGE': ['OKGE'],
    'OPPD': ['OPPD'],
    'SECI': ['SECI'],
    'SPRM': ['SPRM'],
    'SPS': ['SPS'],
    'WAUE': ['WAUE'],
    'WFEC': ['WFEC'],
    'WR': ['WR'],

    'Carolina':['CPLW', 'DUK', 'SCEG', 'SC', 'CPLE']
}

eastern_agg_demand = aggregate_ba_demand(eastern_demand_2016, ba_area)
eastern_agg_demand<jupyter_output>ISNE regional demand was summed from ['ISNE']

NYIS regional demand was summed from ['NYIS']

PJM regional demand was summed from ['OVEC', 'PJM']

AEC regional demand was summed from ['AEC']

SOCO regional demand was summed from ['SOCO']

TVA_LGEE regional demand was summed from ['LGEE', 'TVA']

Florida regional demand was summed from ['NSB', 'HST', 'FPC', 'GVL', 'TEC', 'FMPP', 'SEC', 'TAL', 'JEA', 'FPL']

AECI regional demand was summed from ['AECI']

LRZ1 regional demand was summed from ['LRZ1']

LRZ2_7 regional demand was summed from ['LRZ2_7']

LRZ3_5 regional demand was summed from ['LRZ3_5']

LRZ4 regional demand was summed from ['LRZ4']

LRZ6 regional demand was summed from ['LRZ6']

LRZ8_9_10 regional demand was summed from ['LRZ8_9_10']

CSWS regional demand was summed from ['CSWS', 'SPA']

EDE regional demand was summed from ['EDE']

GRDA regional demand was summed from ['GRDA']

KACY regional demand was summed from ['KACY']

KCPL_INDN regional demand was summed from ['KCPL_I[...]<jupyter_text>5. Detect and Fix OutliersIndentify and fix outliers. Outliers defined as demand hours for which the ramp rate (difference between two hours) is greater than 3 standard deviations of the ramp rates for the BA across all hours in the year. Demand data for hours identified as outliers are replaced with the average of the neighboring non-outlier hour's demand. Stretches of >5 consecutive outlier hours are flagged.<jupyter_code>eastern_fix_outliers = fix_dataframe_outliers(eastern_agg_demand)<jupyter_output>1357 104873.0 104873.0
1358 137033.0 104409.0
1359 103945.0 103945.0
2410 19206.0 19206.0
2411 26343.0 19742.0
2412 20278.0 20278.0
701 23378.0 23378.0
702 4517.0 22913.0
703 22448.0 22448.0
7469 16362.0 16362.0
7470 4106.0 15976.0
7471 15590.0 15590.0
8237 25897.0 25897.0
8238 5649.0 25647.5
8239 25398.0 25398.0
8477 28120.0 28120.0
8478 7081.0 27567.5
8479 27015.0 27015.0
1 31645.0 31645.0
2 22897.0 22897.0
1564 18918.0 18918.0
1565 28308.0 18278.0
1566 17638.0 17638.0
1876 31575.0 31575.0
1877 21080.0 21080.0
3412 33165.0 33165.0
3413 23562.0 23562.0
4850 38735.0 38735.0
4851 27638.0 27638.0
6094 42074.0 42074.0
6095 30860.0 30860.0
6171 30200.0 30200.0
6172 21228.0 21228.0
6266 34131.0 34131.0
6267 24657.0 24657.0
6746 26350.0 26350.0
6747 15276.0 15276.0
7195 30944.0 30944.0
7196 41548.0 31338.5
7197 31733.0 31733.0
7633 27013.0 27013.0
7634 15766.0 15766.0
2746 629.6459464821249 629.6459464821249
2747 484.93109968550647 484.93109968550647
3203 695.4946704359145 695.4946704359145
[...]<jupyter_text>6 Get bus to BA area mapping via countyg via county A. Create File if not AvailableIf the file is not available, it will be created by the `bus_ba_map.ipynb` notebook. Note that it takes around 3h to run.<jupyter_code>path = os.path.join("..", "eastern_demand_v5_demo")
if not os.path.exists(os.path.join(path, "bus_ba_map.csv")):
    %cd path
    %run ./bus_ba_map.ipynb
    %cd -<jupyter_output><empty_output><jupyter_text>B. Read the Files<jupyter_code>bus_map = pd.read_csv(os.path.join(path, "bus_ba_map.csv"), index_col=0)
bus_map_spp = pd.read_csv(os.path.join("spp", "bus_ba_map_spp.csv"), index_col=0)
bus_map_miso = pd.read_csv(os.path.join("miso", "bus_ba_map_miso.csv"), index_col=0)
bus_map_miso['BA'] = bus_map_miso['miso_zone']
bus_map_miso.drop(columns=['miso_zone', 'state'], inplace=True)
no_miso_spp_bus_map = bus_map[~((bus_map['BA'] == 'MISO')|(bus_map['BA'] == 'SWPP'))]
bus_map_new = pd.concat([no_miso_spp_bus_map, bus_map_miso,bus_map_spp]).sort_index()<jupyter_output><empty_output><jupyter_text>Transform to load zone using the demand that has missing data filled and outliers fixed, prepared from the first steps in the demo notebook.<jupyter_code>final_eastern_demand = get_demand_in_loadzone(eastern_fix_outliers, bus_map_new)
final_eastern_demand.index = final_eastern_demand.index.tz_localize(None)
final_eastern_demand
grid = Grid(['Eastern'])
final_eastern_demand.columns = [grid.zone2id[col] for col in final_eastern_demand.columns]
final_eastern_demand = final_eastern_demand.sort_index(axis = 1)
final_eastern_demand
print(final_eastern_demand.columns)
final_eastern_demand.sum(axis=1).plot(figsize=(10,8))
final_eastern_demand.index.name = "UTC Time"
final_eastern_demand.to_csv("eastern_demand_v6.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13035931261933692, 'fraction_numerical': 0.08739802117687902, 'mean_word_length': 5.0775316455696204, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '401029', 'license': 'MIT', 'n_tokens_mistral': 5190, 'n_tokens_neox': 4421, 'n_words': 1115, 'path': 'prereise/gather/demanddata/eia/demo/eastern_demand_v6_demo/eastern_demand_v6_demo.ipynb', 'repo_name': 'lanesmith/PreREISE', 'size': '11522'}