<jupyter_start><jupyter_text>Energy Innovation MCOE Compilation- Setup- Data Outputs    * Plant Level Output    * Unit Level Output    * Export to CSV- Data Components    * Part 1: Basic Plant & Unit Information    * Part 2: Cost Data    * Part 3: Emissions & Public Health Data- Data Validation    * FERC Form 1 vs. EIA    * MCOE Validation -------------  Setup<jupyter_code>%load_ext autoreload
%autoreload 2
# If you're running this notebook on our Jupyterhub, uncomment this line to install the EI MCOE specific code:
# !pip install -e ../
import pudl

import sqlalchemy as sa
import ei_mcoe
from ei_mcoe import OUTPUTS_DIR
from ei_mcoe.ei_mcoe import *
import sys
import os
import matplotlib.pyplot as plt
import logging
from datetime import datetime

from copy import deepcopy
# basic setup for logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(stream=sys.stdout)
formatter = logging.Formatter('%(message)s')
handler.setFormatter(formatter)
logger.handlers = [handler]
pd.options.display.max_columns = None
pudl_settings = pudl.workspace.setup.get_defaults()
pudl_engine = sa.create_engine(pudl_settings['pudl_db'])
# Without these API keys, the analysis will fail.
# You can set them here in the notebook if need be by uncommenting these lines and adding your API keys:
#os.environ['API_KEY_EIA'] = "your-eia-api-key-goes-here"
#os.environ['API_KEY_FRED'] = "your-fred-api-key-goes-here"
assert os.environ.get('API_KEY_EIA')
assert os.environ.get('API_KEY_FRED')
pudl_out = pudl.output.pudltabl.PudlTabl(
    pudl_engine,
    start_date='2019-01-01',
    end_date='2020-12-31',
    freq='AS',
    fill_fuel_cost=True,
    roll_fuel_cost=True,
    fill_net_gen=False,
)<jupyter_output><empty_output><jupyter_text>Data Outputs<jupyter_code>%%time
cems_df = get_cems(pudl_settings, years=[2019, 2020])<jupyter_output><empty_output><jupyter_text>Plant Level Output<jupyter_code>%%time
# beware - this takes several minutes run!!
mcoe_plant = main(pudl_out, cems_df, 'plant-fuel', separate_nems_cols=True, fill_in_fuel_cost=True)
%%time
mcoe_plant[
    (mcoe_plant.report_date.dt.year == NEMS_YEAR)
    & (mcoe_plant.fuel_type_code_pudl == 'coal')
].groupby(['confidence_tier'], dropna=False)[['plant_id_eia']].count()<jupyter_output><empty_output><jupyter_text>Unit Level Output<jupyter_code># unit = main(pudl_out, cems_df, 'unit-fuel', separate_nems_cols=True)<jupyter_output><empty_output><jupyter_text>Export to CSV<jupyter_code>%%time
mcoe_plant_year = mcoe_plant[mcoe_plant.report_year == NEMS_YEAR]
source_df = generate_source_df()

mcoe_plant_year.to_csv(OUTPUTS_DIR / f"mcoe_compilation_{NEMS_YEAR}_{datetime.today().strftime('%Y-%m-%d')}.csv", index=False)
source_df.to_csv(OUTPUTS_DIR / 'mcoe_documentation.csv', index=False)<jupyter_output><empty_output><jupyter_text>-------------------------  Data Components  Part 1: Plant & Unit Level DataEIA-860 and EIA-923 generator-level data is aggregated by either plant or unit and subdivided by broad fuel type (coal, gas, oil, waste). [`Age`] is calculated by finding the weighted average (capacity as weight) of the generators in a given aggregation. [`Heat Rate`] is also calculated by weighted average (this time with net generation as weight). [`MW Nameplate Capacity`] and [`Annual Electricity Net Generation MWh`] at a given level of aggregation are calculated by summing generator-level data. For purely qualitative information (just plant name and location) add [`drop_calcs=True`] to the parameters. Part 2: Cost DataCost and generation data from EIA-860, EIA-923, and FERC Form 1 are subdivided by plant and broad fuel type. The fuel-type breakdown for FERC Form 1 plants is determined by the EIA fuel breakdown for plants of the same PUDL Plant Code. For missing fixed and variable costs from 2018, we've input data from NEMS as a substitute. MCOE is calculated using data from the following sources: NEMS Variable Origins- net_generation_mwh_nems = capacity_factor * 8760 * capacity_mw- fixed_om_18_nems = fixed_om_kw_18_nems * 1000 * capacity_mw- fixed_om_mwh_18_nems = fixed_om_18_nems / net_generation_mwh_nems- variable_om_18_nems = variable_om_mwh_18_nems * net_generation_mwh_nems- fix_var_om_mwh_18_nems = variable_om_mwh_18_nems + fixed_om_kw_18_nems- fixed_v_total_ratio = fixed_om_18_nems / (fixed_om_18_nems + variable_om_18_nems)- var_v_total_ratio = variable_om_18_nems / (fixed_om_18_nems + variable_om_18_nems)- fix_var_om_18_nems =  fixed_om_18_nems + variable_om_18_nems- fix_var_om_mwh_18_nems = fixed_om_mwh_18_nems + variable_om_mwh_18_nems MCOE Variable Origins - total_fuel_cost (Fuel cost) **EIA-923**- capacity_mw (MW Capacity) **EIA-860**- net_generation_mwh (Net MWh Generated) **EIA-923**: - opex_nofuel_ferc1 (Non-Fuel O&M) = **FERC Form 1**: opex_production_total - opex_fuel- fixed_om = fix_var_om * fixed_v_total_ratio- variable_om = fix_var_om * var_v_total_ratio- fixed_om_mwh = fixed_om / net_generation_mwh_ferc1; if null, filled in with fixed_om_mwh_18_nems- fixed_om_mw = fixed_om / capacity_mw- variable_om_mwh = variable_om / net_generation_mwh_ferc1; if null, filled in with variable_om_mwh_18_nems- fix_var_om_mwh = opex_nofuel_ferc1 / net_generation_mwh_ferc1 Data Flags[`Significant Heat Rate Discrepancy`] - a field indicating whether a plant fuel type contains units that have outlier heatrates. If a unit is more than one standard deviation away from the mean value for units of its same fuel type rate, the field will appear [`True`].[`Fixed/Variable O&M used NEMS?`] - a field indicating whether the given row used FERC Form 1 cost data or NEMS cost data. If NEMS were used, the field will appear [`True`]. Part 3: Emissions & Public Health DataCEMS, or Continuous Emission Monitoring Systems, provide detailed information about gas, particulate matter and other pollutants that emanate from various point sources. Here, CEMS data on co2, so2, and nox emissions from generation units is combined with EIA plant data at the plant and unit level, separated by fuel type. Data on PM2.5 emissions comes from Argonne National Laboratory's GREET Model. The model's Electricity Generation Module table 2.1 contains PM2.5 emissions data in g/kwh at the grandularity of NERC region and technology type. The PM2.5 emissions data are mapped onto EIA and CEMS data by creating buckets of the same granularity. --------------  Data Validation  FERC Form 1 vs. EIA The first test looks at the **validity of using EIA fuel percentage values to disaggregate FERC Form 1 data by fuel type.** The following hisograms compare the fuel fractions available in FERC Form 1 with the fuel fractions created by aggregating EIA data by plant and fuel type. The cost factors used in the calculation of MCOE rely on FERC Form 1 fixed and operating cost data broken down by plant and fuel type based on EIA fuel breakdown. To ensure that there is a degree of similarity between the percent breakdown of EIA fuel break down and FERC Form 1 breakdown, this histogram depicts the EIA percent / FERC Form 1 percent. FERC Form 1 has two fuel breakdowns, by MMBtu and by cost. The graph to the left divides EIA percents by FERC Form 1 MMBtu fuel fractions and on the right by cost fractions. The closer the value to 1, the more acurate the comparison.<jupyter_code>#plot_fuel_pct_check(merge_ferc1_eia_fuel_pcts(pudl_out))<jupyter_output><empty_output><jupyter_text>As we know, FERC Form 1 and EIA data don't always match up properly. The following graphs depict **the difference in FERC Form 1 and EIA-860/923 reporting on the these particular values:** [`capacity_mw`], [`opex_fuel`], [`total_mmbtu`], [`net_generation_mwh`], [`capacity_factor`], [`heat_rate_mmbtu_mwh`], [`fuel_cost_per_mwh`], [`fuel_cost_per_mmbtu`], used in the calculation of MCOE.<jupyter_code>plot_eia_v_ferc(pudl_out)<jupyter_output><empty_output><jupyter_text>Heat Rate Comparison<jupyter_code>plot_heat_rate(pudl_out)<jupyter_output><empty_output><jupyter_text>Check the MCOE Compilation this is a check to see the general shape of the fixed and variable O&M of the PUDL data is similar to NEMS<jupyter_code>plant_non_nems = mcoe_plant[~mcoe_plant[['fix_var_is_NEMS']].astype(pd.BooleanDtype()).fix_var_is_NEMS]
#plant_non_nems = plant_19[(~plant_19.fix_var_is_NEMS)]
plt.hist(plant_non_nems.fix_var_om_mwh, density=True, cumulative=True, 
         range=(0,100), label='PUDL (Non-Nems)', alpha=.5,
         bins=100);

plt.hist((mcoe_plant.fix_var_om_mwh_19_nems), density=True, cumulative=True, 
         range=(0,100), label='NEMS', alpha=.7,
         bins=100);
plt.xlabel('Non-Fuel OM $/MWh')
plt.legend()
plt.title("PUDL vs NEMS Non-Fuel OM Distribution")
plt.show()<jupyter_output><empty_output><jupyter_text>The following graph, we're simply plotting the components of MCOE against MCOE.This should be just a straight line... which it appears to be, so yay.The `mcoe` column has the least amount of records in it because it requires having both fuel cost data from EIA and fixed and variable cost data from FERC or NEMS.<jupyter_code>plot_mcoe_vs_nems(mcoe_plant,
                  x_cols=['mcoe'], y_cols=['fix_var_om_mwh','fuel_cost_mwh_eia923'], 
                  log=True, 
                  x_lim=(.1,1e5), y_lim=(.1,1e5),
                  alt_title='MCOE v MCOE components'
                 );<jupyter_output><empty_output><jupyter_text>Now we are getting into comparing the components of MCOE. First let's compare the fixed and variable dollars per MWh.Here are the plots<jupyter_code>plot_mcoe_vs_nems(mcoe_plant, 
                  x_cols=['fix_var_om_mwh'], y_cols=['fix_var_om_mwh_19_nems'], 
                  log=True, 
                  x_lim=(1,1e3), y_lim=(1,1e3),
                  #fuels=['coal','gas']
                 );
plot_mcoe_vs_nems(plant_non_nems, 
                  x_cols=['fix_var_om_mwh'], y_cols=['fix_var_om_mwh_19_nems'], 
                  log=True, 
                  x_lim=(1,1e2), y_lim=(1,1e2),
                  #fuels=['coal','gas']
                 );
plot_mcoe_vs_nems(plant_non_nems, 
                  x_cols=['fixed_om_mwh'], y_cols=['fixed_om_mwh_19_nems'], 
                  log=True, 
                  x_lim=(1,1e3),
                  y_lim=(1,1e3), 
                  #fuels=['coal','gas']
                 );
plot_mcoe_vs_nems(plant_non_nems, 
                  x_cols=['variable_om_mwh'], y_cols=['variable_om_mwh_19_nems'], 
                  log=True, 
                  x_lim=(.01,1e2),
                  y_lim=(.01,1e2)
                 );<jupyter_output><empty_output><jupyter_text>There are the total fixed and variable dollar amounts. They look quite reasonable against NEMS totals.<jupyter_code>plot_mcoe_vs_nems(mcoe_plant, 
                  x_cols=['fix_var_om'], y_cols=['fix_var_om_19_nems'], 
                  log=True, 
                  x_lim=(1e5,1e9),
                  y_lim=(1e5,1e9)
                 );
plot_mcoe_vs_nems(mcoe_plant, 
                  x_cols=['fixed_om'], y_cols=['fixed_om_19_nems'], 
                  log=True, 
                  x_lim=(1e5,1e9),
                  y_lim=(1e5,1e9)
                 );
plot_mcoe_vs_nems(mcoe_plant,
                  x_cols=['variable_om'], y_cols=['variable_om_19_nems'], 
                  log=True, 
                  x_lim=(1e5,1e9),
                  y_lim=(1e5,1e9)
                 );
plot_hist_annual(plant_non_nems, 'mcoe', "MCOE $/MWh")
plot_hist_annual(mcoe_plant, 'fix_var_om_mwh', "Total O&M Costs [$/MWh]")
# this is a brief exploration of the split between fixed and variable costs from NEMS.
# there are clear patterns/differences between coal and gas.
# right now, we aren't using fuel type averages to break out fixed and variable costs
# but we could for the records which don't have associated NEMS data

nems = prep_nems(pudl_out)
fuel_tpyes= nems.fuel_type_code_pudl.unique()
#for fuel_type in fuel_tpyes:
for fuel_type in ['gas','coal']:
    df = nems[nems.fuel_type_code_pudl == fuel_type]
    plt.hist(df.fixed_v_total_ratio, 
             range=(0,1),
             weights=df.capacity_mw, 
             label=f'{fuel_type}',
             bins=40
            )
    plt.title(f"Fixed v Variable Non-Fuel O&M in NEMS for {fuel_type}")
    plt.xlabel("Ratio")
    plt.ylabel("Capacity MW")
    plt.legend(loc='upper left')
    plt.show()
# quick exploration of mcoe
plants_all = mcoe_plant[mcoe_plant.report_year == NEMS_YEAR]
logger.info(f'all {NEMS_YEAR} records:    {len(plants_all)}')
logger.info(f'{NEMS_YEAR} records w/ferc: {len(plants_all[plants_all.fix_var_om_mwh.notnull()])}')
logger.info(f'{NEMS_YEAR} records w/nems: {len(plants_all[plants_all.fix_var_om_mwh_19_nems.notnull()])}')
logger.info(f'{NEMS_YEAR} records w/eia:  {len(plants_all[plants_all.fuel_cost_mwh_eia923.notnull()])}')
logger.info(f'{NEMS_YEAR} records w/moce: {len(plants_all[plants_all.mcoe.notnull()])}')
# quick exploration of capacity
plants_coal_gas = plants_all[plants_all.fuel_type_code_pudl.isin(['coal','gas',])]
logger.info(f'all {NEMS_YEAR} capacity:    {(plants_coal_gas.capacity_mw.sum())/plants_coal_gas.capacity_mw.sum():.01%}')
logger.info(f'{NEMS_YEAR} capacity w/ferc: {(plants_coal_gas[plants_coal_gas.fix_var_om_mwh.notnull()].capacity_mw.sum())/plants_coal_gas.capacity_mw.sum():.01%}')
logger.info(f'{NEMS_YEAR} capacity w/nems: {(plants_coal_gas[plants_coal_gas.fix_var_om_mwh_19_nems.notnull()].capacity_mw.sum())/plants_coal_gas.capacity_mw.sum():.01%}')
logger.info(f'{NEMS_YEAR} capacity w/eia:  {(plants_coal_gas[plants_coal_gas.fuel_cost_mwh_eia923.notnull()].capacity_mw.sum())/plants_coal_gas.capacity_mw.sum():.01%}')
logger.info(f'{NEMS_YEAR} capacity w/mcoe: {(plants_coal_gas[plants_coal_gas.mcoe.notnull()].capacity_mw.sum())/plants_coal_gas.capacity_mw.sum():.01%}')
# capacity coverage
plants_all[plants_all.mcoe.notnull()].capacity_mw.sum()/plants_all[plants_all.fuel_type_code_pudl.isin(['coal','gas'])].capacity_mw.sum()
# net generation coverage
plants_all[plants_all.mcoe.notnull()].net_generation_mwh.sum()/plants_all[plants_all.fuel_type_code_pudl.isin(['coal','gas'])].net_generation_mwh.sum()
# coal capacity coverage
(plants_all[(plants_all.mcoe.notnull())
          & plants_all.fuel_type_code_pudl.isin(['coal'])
         ].capacity_mw.sum()
 /plants_all[(plants_all.fuel_type_code_pudl.isin(['coal']))].capacity_mw.sum())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10684683429685868, 'fraction_numerical': 0.018597199972139027, 'mean_word_length': 4.397744360902256, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '436383', 'license': 'MIT', 'n_tokens_mistral': 5395, 'n_tokens_neox': 4941, 'n_words': 1223, 'path': 'notebooks/mcoe_compilation.ipynb', 'repo_name': 'catalyst-cooperative/ei-mcoe-map', 'size': '14357'}