<jupyter_start><jupyter_text>Introduction to CrunchersIn this notebook we demonstrate crunchers, silicone's most fundamental infilling methods, using one example. In the next notebook (`01_Prefiltering`), we will discuss the importance of selecting the most relevant data before infilling. In later notebooks we will uncover more crunchers, and in notebook `05_Multiple_infillers`, we will discuss tools to ensure consistency when applying crunchers to infill multiple variables, or where some information is already known. Notebook 06 details how to project forwards an emission trajectory. In order to run this notebook, you will need to install silicone from as described in the readme in order to run this code. It introduces one specific cruncher, the 'closest RMS' cruncher, and the basic tools to manipulate it.   Imports<jupyter_code>import os.path
import traceback

import pandas as pd
import pyam
import matplotlib.pyplot as plt
import numpy as np

import silicone.database_crunchers
from silicone.utils import download_or_load_sr15<jupyter_output><empty_output><jupyter_text>Assembling example dataHere we pull some example data by downloading it from the IAMC 1.5C Scenario Explorer and Data hosted by IIASA (for details, see Huppman et al 2019, Integrated Assessment Modeling Consortium & International Institute for Applied Systems Analysis, 2018). If the data has already been downloaded before, we will use that instead for brevity, handled by a convenient wrapper function. We select only a few cases.<jupyter_code>valid_model_ids = [
    "MESSAGE*",
    "AIM*",
    "GCAM*",
    "IMAGE*",
    "WITCH*",
    # "IEA*",
    # "C-ROADS*",
    # "MERGE*",
    # "POLES*",
    # "REMIND*",
]
SR15_SCENARIOS = "./sr15_scenarios.csv"
sr15_data = download_or_load_sr15(SR15_SCENARIOS, valid_model_ids)<jupyter_output>pyam - INFO: Running in a notebook, setting up a basic logging at level INFO
pyam.core - INFO: Reading file sr15_scenarios.csv<jupyter_text>Starting pointOur starting point is the test data, loaded as a `pyam.IamDataFrame`. It may be helpful for understanding some of this tutorial for you to know more about IamDataFrames, documented at https://pyam-iamc.readthedocs.io/en/latest/. The key functions are `.filter()`, which selects a subset of data, and `.timeseries()`, which restructures the data into columns by date, with long indexes containing information other than the value.<jupyter_code>sr15_data = pyam.IamDataFrame(SR15_SCENARIOS)
sr15_data.timeseries().head()<jupyter_output>pyam.core - INFO: Reading file sr15_scenarios.csv<jupyter_text>CrunchersSilicone's 'crunchers' are used to determine the relationship between a 'follower variable' and 'lead variable(s)' from a given database. The 'follower variable' is the variable for which we want to generate data e.g. `Emissions|C3F8` while the 'lead variable(s)' is the variable we want to use in order to infer a timeseries of the 'follower variable'. For most crunchers, the lead variable is currently a list with only a single item allowed, e.g. `[Emissions|HFC]` but for RMS closest cruncher (and other crunchers in the future) several elements may be included. The follower is always a single string - to infill multiple values in a collective way (e.g. splitting HFC into its various components) see Multiple Infillers. Each cruncher needs to be initialised with a database and has a `derive_relationship` method, which returns the infilling function. Its docstring describes what it does.<jupyter_code>print(silicone.database_crunchers.base._DatabaseCruncher.derive_relationship.__doc__)
# # an alternative, pop-up interface is shown if you uncomment the line below
# silicone.database_crunchers.base._DatabaseCruncher.derive_relationship?<jupyter_output>Derive the relationship between two variables from the database

        Parameters
        ----------
        variable_follower : str
            The variable for which we want to calculate timeseries (e.g.
            ``"Emissions|C5F12"``).

        variable_leaders : list[str]
            The variables we want to use in order to infer timeseries of
            ``variable_follower`` (e.g. ``["Emissions|CO2", "Emissions|CH4"]``)

        **kwargs
            Keyword arguments used by this class to derive the relationship between
            ``variable_follower`` and ``variable_leaders``.

        Returns
        -------
        :obj:`func`
            Function which takes a :obj:`pyam.IamDataFrame` containing
            ``variable_leaders`` timeseries and returns timeseries for
            ``variable_follower`` based on the derived relationship between the two.
            Please see the source code for the exact definition (and docstring) of the
            returned functi[...]<jupyter_text>These crunchers are best explored by looking at the examples below. Closest Root Mean Square cruncherThis cruncher finds the scenario and model in the infiller database that minimises the RMS distance between the the lead emissions in the infiller and infillee data. The follow data is then directly taken from that scenario. The full documentation is as follows:<jupyter_code>print(silicone.database_crunchers.RMSClosest.__doc__)<jupyter_output>Database cruncher which uses the 'closest RMS' technkque.

    This cruncher derives the relationship between two or more variables by finding the
    scenario which has the most similar timeseries for the lead gases in the database.
    The follower gas timeseries is then simply copied from the closest scenario.

    Here, 'most similar' is defined as the smallest time-averaged root mean squared (L2)
    difference. If multiple lead values are used, they may be weighted differently to
    account for differences between the reported units. The most similar model/scenario
    combination minimises

    .. math::
        RMS error = \sum_l w_l \left ( \frac{1}{n} \sum_{t=0}^n (E_l(t) - e_l(t))^2 \right )^{1/2}

    where :math:`l` is a lead gas, :math:`w_l` is a weighting for that lead gas,
    :math:`n` is the total number of timesteps in all lead gas timeseries,
    :math:`E_l(t)` is the lead gas emissions timeseries and :math:`e_l(t)` is a lead
    gas emissions timeseries in th[...]<jupyter_text>InfillingFirstly, let's cut the database down to a size that is comprehensible.<jupyter_code>sr15_data_closest_rms = sr15_data.filter(model=["WITCH-GLOBIOM 4.2"])
sr15_data_closest_rms.head()<jupyter_output><empty_output><jupyter_text>Next, we initialise the type of cruncher we want with the infiller database.<jupyter_code>cruncher = silicone.database_crunchers.RMSClosest(sr15_data_closest_rms)<jupyter_output><empty_output><jupyter_text>Now we can derive the relationship between the follower and leader, here `Emissions|CO2` and `Emissions|VOC` respectively. This returns a infiller function, which we will use later. Firstly we will check out the docstring.<jupyter_code>filler = cruncher.derive_relationship("Emissions|VOC", ["Emissions|CO2"])
print(filler.__doc__)
# filler info:<jupyter_output>Filler function derived from :obj:`RMSClosest`.

            Parameters
            ----------
            in_iamdf : :obj:`pyam.IamDataFrame`
                Input data to fill data in

            Returns
            -------
            :obj:`pyam.IamDataFrame`
                Filled in data (without original source data)

            Raises
            ------
            ValueError
                If there are any inconsistencies between the timeseries, units or
                expectations of the program and ``in_iamdf``, compared to the database
                used to generate this ``filler`` function.<jupyter_text>Now we can use this relationship to do some infilling. As a sanity check, we firstly make sure that if we pass in a CO$_2$ timeseries which is already in the database, we get back its VOC emissions timeseries.<jupyter_code>example_model_scen = {
    "model": "WITCH-GLOBIOM 4.2",
    "scenario": "ADVANCE_INDC",
}
example_input = sr15_data_closest_rms.filter(**example_model_scen).data
example_input["model"] = "example"
example_input["scenario"] = "example"
example_input = pyam.IamDataFrame(example_input)<jupyter_output><empty_output><jupyter_text>We can now use the function we returned above to infill the example data, and see that we recover the data we put in.<jupyter_code>example_input_filled = filler(example_input)
example_input_filled.timeseries()
sr15_data_closest_rms.filter(
    variable="Emissions|VOC", **example_model_scen
).timeseries()<jupyter_output><empty_output><jupyter_text>As expected, the cruncher has picked out the scenario which matches and returned its VOC timeseries. This can also be seen in a plot below, where the dashed (infilled) line perfectly matches the pink line.<jupyter_code>pkwargs = {"color": "scenario", "linewidth": 2}
target_kwargs = {"color": "black", "linestyle": "--", "dashes": (10, 15)}
fig = plt.figure(figsize=(16, 9))
ax = fig.add_subplot(121)
sr15_data_closest_rms.filter(variable="*CO2").plot(ax=ax, **pkwargs)
example_input.filter(variable="*CO2").plot(ax=ax, label="To infill", **target_kwargs)
ax.set_title("Database CO$_2$ and lead emissions timeseries")

ax = fig.add_subplot(122)
sr15_data_closest_rms.filter(variable="*VOC").plot(ax=ax, **pkwargs)
example_input_filled.filter(variable="*VOC").plot(
    ax=ax, label="Infilled", **target_kwargs
)
ax.set_title("Database VOC and filled follower emissions timeseries")

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Now we can use our filler to infill other timeseries not present in our infiller database.<jupyter_code>filler_input = sr15_data.filter(
    model="MESSAGEix-GLOBIOM 1.0", scenario="CD-LINKS_NPi2020_1600"
)
# This scenario is missing `Emissions|VOC`.
filler_input.filter(variable="Emissions|VOC").data
# It can go in the same infiller we defined previously.
filler_input_filled = filler(filler_input)
fig = plt.figure(figsize=(16, 9))
ax = fig.add_subplot(121)
sr15_data_closest_rms.filter(variable="*CO2").plot(ax=ax, **pkwargs)
filler_input.filter(variable="*CO2").plot(label="To infill", ax=ax, **target_kwargs)
ax.set_title("Database CO$_2$ and lead emissions timeseries")

ax = fig.add_subplot(122)
sr15_data_closest_rms.filter(variable="*VOC").plot(ax=ax, **pkwargs)
filler_input_filled.filter(variable="*VOC").plot(
    ax=ax, label="Infilled", **target_kwargs
)
ax.set_title("Database VOC and filled follower emissions timeseries")

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>As we can see, on an RMS basis our input timeseries is closest to the 'ADVANCE_2030_Med2C' scenario and hence its `Emissions|VOC` pathway is returned.<jupyter_code>filler_input_filled.timeseries()
sr15_data_closest_rms.filter(
    scenario="ADVANCE_2030_Med2C", variable="Emissions|VOC"
).timeseries()<jupyter_output><empty_output><jupyter_text>This may be regarded as conceptually the simplest of the data-based crunchers. The other crunchers are introduced in other notebooks. For more complicated applications, this cruncher may be used with multiple leads, and optionally weighting them differently. For instance, we might want the closest scenario considering both CO2 and CH4. Be careful when using this feature as it will directly use the values in whatever units the variables are reported in. Since CO2 emissions are typically larger than CH4, CH4 is therefore underemphasised in this approach. We will use the IMAGE model family for this demonstration.<jupyter_code>follow = "Emissions|BC"
leads = ["Emissions|CO2|AFOLU", "Emissions|CH4"]
multi_filler = cruncher.derive_relationship(follow, leads)
sr15_data_multi = sr15_data.filter(model="IMAGE*")
multi_filler_input = sr15_data.filter(
    model="MESSAGEix-GLOBIOM 1.0", scenario="CD-LINKS_NoPolicy"
)
multi_cruncher = silicone.database_crunchers.RMSClosest(sr15_data_multi)
multi_filler = multi_cruncher.derive_relationship(follow, leads)
multi_results = multi_filler(multi_filler_input)<jupyter_output><empty_output><jupyter_text>However it might be desired that the relative importance of different emissions was determined by the GWP100 factors instead of the mass units that emissions are reported in. In this case, we need to add a dictionary of the relative importance.<jupyter_code>multi_GWP100_filler = multi_cruncher.derive_relationship(
    follow, leads, {"Emissions|CO2|AFOLU": 1, "Emissions|CH4": 28}
)
multi_GWP100_results = multi_GWP100_filler(multi_filler_input)
pkwargs = {"color": "scenario", "linewidth": 1}
fig = plt.figure(figsize=(16, 9))
ax = fig.add_subplot(131)
sr15_data_multi.filter(variable="Emissions|CH4").plot(
    ax=ax, label="_", legend=None, **pkwargs
)
multi_filler_input.filter(variable="Emissions|CH4").plot(
    ax=ax, label="target", **target_kwargs
)
ax.set_title("CH4 lead comparison")
ax = fig.add_subplot(132)
sr15_data_multi.filter(variable="Emissions|CO2|AFOLU").plot(ax=ax, **pkwargs)
multi_filler_input.filter(variable="Emissions|CO2|AFOLU").plot(
    ax=ax, **target_kwargs, label="_"
)
ax.set_title("CO2 lead comparison")
ax = fig.add_subplot(133)
sr15_data_multi.filter(variable=follow).plot(ax=ax, **pkwargs, label="_")
multi_results.plot(ax=ax, **target_kwargs, label="Infilled with mass units")
multi_GWP100_results.plot(
    ax=ax,
    linestyle="--",
    dashes=(10, 10),
    color="blue",
    label="Infilled with GWP100 units",
)
ax.set_title("Infilled BC comparison")<jupyter_output>pyam.plotting - INFO: >=13 labels, not applying legend
pyam.plotting - INFO: >=13 labels, not applying legend
{'fraction_non_alphanumeric': 0.0915414785761859, 'fraction_numerical': 0.015096573669799452, 'mean_word_length': 4.979646017699115, 'pattern_counts': {'":': 11, '<': 55, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '323855', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4256, 'n_tokens_neox': 4025, 'n_words': 1352, 'path': 'notebooks/00_Introductory_infilling.ipynb', 'repo_name': 'znicholls/silicone', 'size': '13513'}