<jupyter_start><jupyter_text>Westeros Tutorial - Introducing `historical_new_capacity`  Scope of this tutorialThis tutorial takes a closer look at the parameter [`historical_new_capacity`](https://docs.messageix.org/en/stable/model/MESSAGE/parameter_def.html?highlight=historical_new_capacityhistorical-capacity-and-activity-values).We will take a look at how MESSAGEix defines historical periods, and how to apply and parametrize one of the historical parameters, `historical_new_capacity`.We will further investigate the concept of early retirement i.e. the retirement of a powerplant before it reaches the end of its lifetime.**Pre-requisites**- You have the *MESSAGEix* framework installed and working- You have run Westeros baseline scenario (``westeros_baseline.ipynb``) and solved it successfullyIn the first part of the tutorial, we will load and subsequently clone the Westeros baseline scenario.The new scenario will be simplified in order to demonstrate how to parametrize `historical_new_capacity` for `coal_ppl`.In the second part, we will then add a cheaper electricity generation technology, `gas_ppl`, as shown in the figure below.  The `gas_ppl` will  demonstrate the early retirement of `coal_ppl`.Lastly, we will look at how to circumvent early retirement for a specific technology, should this be undesired. Online documentationThe documentation of the MESSAGEix framework is available at [https://docs.messageix.org](https://docs.messageix.org) As in the "original" tutorial, we start by importing all the packages we need.<jupyter_code>import pandas as pd
import ixmp
import message_ix

from message_ix.utils import make_df

%matplotlib inline
# Loading Modeling platform
mp = ixmp.Platform()<jupyter_output><empty_output><jupyter_text>Load and clone Westeros baseline scenario<jupyter_code>base = message_ix.Scenario(mp, model="Westeros Electrified", scenario="baseline")
scenario = base.clone(model="Westeros Electrified", scenario="baseline_historic_new_capacity", keep_solution=False)<jupyter_output><empty_output><jupyter_text>Define generic parameters required for the example<jupyter_code>country = "Westeros"
model_horizon = [700, 710, 720]<jupyter_output><empty_output><jupyter_text>A quick look at historical parameters and their purpose"Historical parameters" can only be set for "historical time-periods".The "historical time-periods" are all the years which have been defined prior to the `firstmodelyear`.These can be used for two purposes.1. For **calibration of the model**: The `firstmodelyear` is used to specify as of which time-period the "optimization" should be carried out.There will be some past dependencies though, such as technology investments, their activity or emissions amongst others.In order to account for these dependencies, we can use the parameters "historical_new_capacity", "historical_capacity" and "historical_emissions" respectively.These will have implications on dynamic constraints on capacities or activity.In the case of emissions, `historical_emissions` can be accounted for in cumulative emission bounds.2. The second application is when running **complex scenario setups**.If for example the aim is to assess climate mitigation efforts based on achieving the NDCs in 2030, then two scenarios can be run.Scenario a) implements "only" the NDCs related policies up to 2030 (see dashed orange line *\"NDC\"* in the panel a. of the figure below).After 2030, there is no continuation of policies assumed, hence the model will return to a "baseline" trajectory (see solid blue line *\"Baseline\"* in the panel a. of the figure below).This scenario will then serve as a basis for further analysis - e.g. scenario b) (see red dashed line *\"NDC-mitigation\"* in panel b. in the figure below).In order to avoid an "overachievement" of the NDCs, the time-periods until 2030 should be *fixed*.The historical time-periods (see the extension of the grey block in panel b. in the figure below) are therefore shifted.This can be done in the process of cloning the NDC scenario for use with mitigation runs.In the process of [cloning](https://docs.messageix.org/en/stable/api.html?highlight=clonemessage_ix.Scenario.clone) a scenario, the argument `shift_first_model_year=` with the corresponding time-period can be passed.In the process of cloning the scenario, for the example below, all time-periods prior to 2035 will be set to "historical" time-periods and its solutions will be transferred to historical_activity and historical_new_capacity. So the results ("VAR" - solved variables e.g. the activity-level) for those years will be added to the corresponding "historical" parameters.      Historical time-periods in our scenario<jupyter_code>years = scenario.set('year').tolist()
print(f"The model years include: {years}")
print(f"the `firstmodelyear` is {scenario.firstmodelyear}")<jupyter_output><empty_output><jupyter_text>Define generic functions<jupyter_code># Retrieve historic time-steps
history = [y for y in years if y < scenario.firstmodelyear]

# Helper df to define `historical_new_capacity`
base_capacity = {
    'node_loc': country,
    'year_vtg': history,
    'unit': 'GWa',
}<jupyter_output><empty_output><jupyter_text>Part 1.: Parametrizing `historical_new_capacity`We are going to undertake a few simplications of the baseline scenario to demonstrate how to set the parameter "historical_new_capacity".- Step 1.1: Scenario simplifications.  - We will reduce the example to a single technology, by removing `wind_ppl`.  - We will change the `demand` to remain constant.- Step 1.2: Scenario adjustments to account for simplifications.  - We will remove the growth constraint for `coal_ppl`.  - We will reparametrize "historical_new_capacity" so that `coal_ppl` can meet demand in the `firstmodelyear`.  - We will adjust the technical lifetime of `coal_ppl`.  Step 1.1.: Scenario simplifications  Remove `wind_ppl`First, we are going to remove the technology 'wind_ppl'.Later on in the tutorial we are instead going to add a more efficient gas powerplant.We do this, because we want to introduce a technology, which will result in the early retirement of the current coal_ppl.<jupyter_code>scenario.check_out()
scenario.remove_set('technology', 'wind_ppl')<jupyter_output><empty_output><jupyter_text>Adjust `demand` Next we will modify the demand to remain constant, hence to avoid the requirement in the optimization time-period to build any new capacity<jupyter_code>df = scenario.par('demand')
df['value'] = float(df[df['year'] == 700]['value'])
scenario.add_par('demand', df)
df<jupyter_output><empty_output><jupyter_text>Step 1.2: Adjust `coal_ppl`  Remove dynamic growth constraintsNow, we will remove the growth constraint on the current coal_ppl.<jupyter_code>df = scenario.par('growth_activity_up', filters={'technology': 'coal_ppl'})
scenario.remove_par('growth_activity_up', df)<jupyter_output><empty_output><jupyter_text>Reparametrize `historical_new_capacity`Now, the `historical_new_capacity` parameter of `coal_ppl` will be adjusted so that sufficient capacity is installed in 690, to meet demand in the `firstmodelyear`.- Retrieve demand of `firstmodelyear`.- Account for grid losses. As both the `capacity_factor` and efficicency for `coal_ppl` are one, these need not be accounted for.- Account for duration of the time-period, as the parameter "historical_new_capacity" is an annual value.<jupyter_code># We assume demand is constant. We will hence start by retrieving the demand in the `firstmodelyear`,
# and assume this also applies for historic time-periods.
demand_of_firstmodelyear= float(scenario.par('demand', filters={'year': scenario.firstmodelyear})['value'])
print(f"The demand of the `firstmodelyear` is {demand_of_firstmodelyear} GWa.")
# We know that there are loses that occur when electricity is transmitted via the `grid`.
grid_eff = float(scenario.par('output', filters={'year_vtg': scenario.firstmodelyear,
                                                 'year_act': scenario.firstmodelyear,
                                                 'technology': 'grid'})['value'])
print(f"Grid efficiency is {grid_eff}.")
demand_of_firstmodelyear/= grid_eff
duration_period = float(scenario.par('duration_period', filters={'year': history})['value'])
print(f"The duration of the period is {duration_period} years.")
historical_new_capacity = demand_of_firstmodelyear / duration_period<jupyter_output><empty_output><jupyter_text>Each year within the historical 10 year period, a 11,11 GW `coal_ppl` must be added to meet the demand of the `firstmodelyear`. Therefore, the value of `historical_new_capacity` is calculated to `historical_new_capacity = demand_of_firstmodelyear / duration_period`.<jupyter_code>df = make_df(base_capacity, technology='coal_ppl', value=historical_new_capacity)
print(f"The `historical_new_capacity` for `coal_ppl` is set to {round(historical_new_capacity, 2)}")
scenario.add_par('historical_new_capacity', df)<jupyter_output><empty_output><jupyter_text>Adjust technical lifetimeWe need to ensure the parameter is also defined for historical vintages.<jupyter_code>df = scenario.par('technical_lifetime', filters={'technology': 'coal_ppl', 'year_vtg': scenario.firstmodelyear})
df['year_vtg'] = history[0]
scenario.add_par('technical_lifetime', df)<jupyter_output><empty_output><jupyter_text>Time to Solve the Model<jupyter_code>scenario.commit('Adjusted baseline to demonstrate parameter `historical_new_capacity`')
scenario.set_as_default()
scenario.solve()<jupyter_output><empty_output><jupyter_text>Plotting Results<jupyter_code>from message_ix.reporting import Reporter
from message_ix.util.tutorial import prepare_plots

rep = Reporter.from_scenario(scenario)
prepare_plots(rep)<jupyter_output><empty_output><jupyter_text>Capacity<jupyter_code>rep.set_filters(t=["coal_ppl"])
rep.get("plot capacity")<jupyter_output><empty_output><jupyter_text>For the optimization time-periods, we can see from the above figure that `coal_ppl` capacity remains constant.     **New capacity installations vs. Total installed capacity**In the figure below we have illustrated the correlation between new capacity installations (left hand side) and total installed capacity.The corresponding scenario results can be retrieved executing `scenario.var("CAP_NEW")` and `scenario.var("CAP")` respectively.As can be seen in the left hand panel, new capacity is installed at two points in time.The first occurs in a *historical* time-period, in the year 690.The investment costs for these capacities lie outside the optimization time-frame.Therefore these do not contribute to the objective function.The powerplant in our example has a `technical_lifetime` of 20 years.The capacity installed in 690 will therefore need to be replaced in 710.The investment costs for this capacity addition is therefore accounted for in 710.    In the righthand panel, we can see the corresponding total installed capacity.The `duration_period` of the time-steps in our example are always 10 years.In order to determine the total installed capacity, we can multiply the new installed capacity, which is an annual value, by 10.         **NOTE: that if the technical lifetime is such that it covers a time-period only partially, then the total installed capacity shown in the variable "CAP", will only reflect the number of years covered.**  Part 2.: Exploring early retirement.In order to demonstrate early retirement of technologies, we are going to add a new, cheaper electricity generation technology.We will then look at what parameters can be used to *force* the model to use *non-economical* technology options.- Step 2.1: Add technology `gas_ppl`.  - We will add the technology `gas_ppl` to the set `technology`.  - We will add parameters `output`, `capacity_factor`, `technical_lifetime`, `inv_cost` and `var_cost` (the parameters are listed in the table below).| Parameter  | `coal_ppl` | `gas_ppl` || ---------- | ---------- | --------- || Efficiency \[%\] | 100 | 100 || Capacity factor \[%\] | 100 | 100 || Technical lifetime \[years\] | 20 | 40 || Investment costs \[USD/kW\] | 500 | 300 || Variable costs \[USD/kWa\] | 30 | 10 |  - Step 2.2: Explore parameters that can force the use of `coal_ppl`, despite being uneconomical.  - `bound_capacity_lo`<jupyter_code>scenario2 = scenario.clone(model='Westeros Electrified', scenario='baseline_historic_new_capacity_part2', keep_solution=False)
scenario2.check_out()<jupyter_output><empty_output><jupyter_text>Define generic Functions<jupyter_code>year_df = scenario2.vintage_and_active_years()
vintage_years, act_years = year_df['year_vtg'], year_df['year_act']

base = {
    'node_loc': country,
    'year_vtg': vintage_years,
    'year_act': act_years,
    'mode': 'standard',
    'time': 'year',
    'unit': '-',
}

base_output = make_df(base, node_dest=country, time_dest='year')

base_capacity_factor = {
    'node_loc': country,
    'year_vtg': vintage_years,
    'year_act': act_years,
    'time': 'year',
    'unit': '-',
}

base_technical_lifetime = {
    'node_loc': country,
    'year_vtg': model_horizon,
    'unit': 'y',
}

base_inv_cost = {
    'node_loc': country,
    'year_vtg': model_horizon,
    'unit': 'USD/kW',
}

base_var_cost = {
    'node_loc': country,
    'year_vtg': vintage_years,
    'year_act': act_years,
    'mode': 'standard',
    'time': 'year',
    'unit': 'USD/kWa',
}<jupyter_output><empty_output><jupyter_text>Step 2.1: Add `gas_ppl`  Define a new technology `gas_ppl`<jupyter_code>scenario2.add_set("technology", ['gas_ppl'])<jupyter_output><empty_output><jupyter_text>Parmetrize `output`<jupyter_code>gas_out = make_df(base_output, technology='gas_ppl', commodity='electricity', 
                   level='secondary', value=1., unit="GWa")
scenario2.add_par('output', gas_out)<jupyter_output><empty_output><jupyter_text>Parametrize `capacity_factor`<jupyter_code>capacity_factor = {
    'gas_ppl': 1,
}

for tec, val in capacity_factor.items():
    df = make_df(base_capacity_factor, technology=tec, value=val)
    scenario2.add_par('capacity_factor', df)<jupyter_output><empty_output><jupyter_text>Parametrize `technical_lifetime`<jupyter_code>lifetime = {
    'gas_ppl': 40,
}

for tec, val in lifetime.items():
    df = make_df(base_technical_lifetime, technology=tec, value=val)
    scenario2.add_par('technical_lifetime', df)<jupyter_output><empty_output><jupyter_text>Parametrize `inv_cost`<jupyter_code>costs = {
    'gas_ppl': 300,
}

for tec, val in costs.items():
    df = make_df(base_inv_cost, technology=tec, value=val)
    scenario2.add_par('inv_cost', df)<jupyter_output><empty_output><jupyter_text>Parametrize `var_cost`<jupyter_code>costs = {
    'gas_ppl': 10,
}

for tec, val in costs.items():
    df = make_df(base_var_cost, technology=tec, value=val)
    scenario2.add_par('var_cost', df)<jupyter_output><empty_output><jupyter_text>Time to Solve the Model<jupyter_code>scenario2.commit('')
scenario2.set_as_default()
scenario2.solve()<jupyter_output><empty_output><jupyter_text>Plotting Results<jupyter_code>rep2 = Reporter.from_scenario(scenario2)
prepare_plots(rep2)<jupyter_output><empty_output><jupyter_text>CapacityWe will first plot the capacity of our first scenario and then compare these to the capacity installations for the exmple for which we added the cheaper electricity generation option, `gas_ppl`.<jupyter_code>rep.get("plot capacity")
rep2.set_filters(t=["coal_ppl", "gas_ppl"])
rep2.get("plot capacity")<jupyter_output><empty_output><jupyter_text>We can see from the two plots, that in the example for which we have added the cheaper electricity generation technology, `gas_ppl`, `coal_ppl` is retired in the `firstmodelyear`. Further details can be seen by looking at the variable `CAP`.<jupyter_code>scenario2.var('CAP', filters={'technology': ['coal_ppl', 'gas_ppl']})<jupyter_output><empty_output><jupyter_text>Step 2.2: Using bounds to force the model to use `coal_ppl`There are multiple bounds that can be used to *force* the model to use the historical capacity of `coal_ppl` until the end of its lifetime.- `bound_total_capacity_lo` (possibly in combination with `bound_total_capacity_up`).- `bound_activity_lo` (possibly in combination with `bound_activity_up`).- dynamic constraints on activity and/or capacity.- `fixed_activity` or `fixed_capacity` can also be used.<jupyter_code>scenario2.remove_solution()
scenario2.check_out()<jupyter_output><empty_output><jupyter_text>`bound_total_capacity_lo`Add lower bound on capacity for 'coal_ppl' in `year_act` installed in 690.In order to calculate the `bound_capacity_lo`, we first need to retrieve the `historical_new_capacity`. As this is an annual value, we will need to account for the `duration_period` when defining the `bound_capacity_lo`.**NOTE: That `bound_capacity_lo` is indexed over the activity years, so if there are multiple vintages installed in historical years, their individual `technical_lifetime` will need to be accounted for when formualting the constraint.**<jupyter_code># Retrieve `historical_new_capacity`
value = float(scenario2.par('historical_new_capacity',
                            filters={'technology': 'coal_ppl'}).value)

# Retrieve `duration_period` for the year 700
duration_period = float(scenario.par('duration_period',
                                     filters={'year': scenario.firstmodelyear})['value']) 

value *= duration_period

df = pd.DataFrame({'node_loc': country,
                   'technology': 'coal_ppl',
                   'year_act': [700],
                   'value': value})
scenario2.add_par('bound_total_capacity_lo', df)
print(f"The `bound_total_capacity_lo` for `coal_ppl` is set to {round(value, 2)} for the year 700.")<jupyter_output><empty_output><jupyter_text>Time to Solve the Model<jupyter_code>scenario2.commit('')
scenario2.set_as_default()
scenario2.solve()<jupyter_output><empty_output><jupyter_text>Plotting Results<jupyter_code>rep2 = Reporter.from_scenario(scenario2)
prepare_plots(rep2)<jupyter_output><empty_output><jupyter_text>CapacityWhen plotting capacity, we can see that the bound has in fact achieved the desired effect, but only partialy. While the capacity of the `coal_ppl` is maintained, we can also observe that the capacity for `gas_ppl` is also built in the `firstmodelyear`. This can be caused for example, because there are no costs associated with maintaining the capacity of `coal_ppl` i.e. as we haven't defined fixed operating and maintance costs.<jupyter_code>rep2.set_filters(t=["coal_ppl", "gas_ppl"])
rep2.get("plot capacity")<jupyter_output><empty_output><jupyter_text>ActivityLooking at the results for activity, we can see that `coal_ppl` is no longer generating electricity.<jupyter_code>rep2.get("plot activity")<jupyter_output><empty_output><jupyter_text>so, an alternative would be to use the "bound_activity_lo".<jupyter_code>scenario2.remove_solution()
scenario2.check_out()<jupyter_output><empty_output><jupyter_text>`bound_activity_lo`Add lower bound on activity for 'coal_ppl' in `year_act` installed in 690.In order to calculate the `bound_activity_lo`, we first need to retrieve the `historical_new_capacity`.As this is an annual value, we will need to account for the `duration_period` when defining the `bound_activity_lo`.We would normally also need to account for the `capacity_factor` as well as the efficiency, but we know that there are both 1.**NOTE: That `bound_activity_lo` is indexed over the activity years, so if there are multiple vintages installed in historical years, their individual `technical_lifetime` will need to be accounted for when formualting the constraint.**<jupyter_code># Retrieve `historical_new_capacity`
value = float(scenario2.par('historical_new_capacity',
                            filters={'technology': 'coal_ppl'}).value)

# Retrieve `duration_period` for the year 700
duration_period = float(scenario.par('duration_period',
                                     filters={'year': scenario.firstmodelyear})['value']) 

value *= duration_period

df = pd.DataFrame({'node_loc': country,
                   'technology': 'coal_ppl',
                   'year_act': [700],
                   'value': value,
                   'mode': 'standard',
                   'time': 'year',
                   'units': 'GWa'})
scenario2.add_par('bound_activity_lo', df)
print(f"The `bound_activity_lo` for `coal_ppl` is set to {round(value, 2)} for the year 700.")<jupyter_output><empty_output><jupyter_text>As we are using the same scenario as before, we will need to remove the `bound_total_capacity_lo` which we previously introduced.<jupyter_code>df = scenario2.par('bound_total_capacity_lo')
scenario2.remove_par('bound_total_capacity_lo', df)<jupyter_output><empty_output><jupyter_text>Time to Solve the Model<jupyter_code>scenario2.commit('')
scenario2.set_as_default()
scenario2.solve()<jupyter_output><empty_output><jupyter_text>Plotting Results<jupyter_code>rep2 = Reporter.from_scenario(scenario2)
prepare_plots(rep2)<jupyter_output><empty_output><jupyter_text>CapacityIf we now replot capacity, we can see that capacity for `gas_ppl` is only added of 710.<jupyter_code>rep2.set_filters(t=["coal_ppl", "gas_ppl"])
rep2.get("plot capacity")<jupyter_output><empty_output><jupyter_text>ActivityLook at the results for activity, we can see that `coal_ppl` generating electricity up until the year 700.<jupyter_code>rep2.get("plot activity")
mp.close_db()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09605336816570256, 'fraction_numerical': 0.009143496921067364, 'mean_word_length': 5.779569892473118, 'pattern_counts': {'":': 0, '<': 158, '<?xml version=': 0, '>': 157, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '325744', 'license': 'Apache-2.0', 'n_tokens_mistral': 6674, 'n_tokens_neox': 6298, 'n_words': 2062, 'path': 'tutorial/westeros/westeros_historical_new_capacity.ipynb', 'repo_name': 'LauWien/message_ix', 'size': '21436'}