<jupyter_start><jupyter_text>![image](https://github.com/eWaterCycle/ewatercycle/raw/main/docs/examples/logo.png)  Case study 1: Hello world. Generate forcings and run Marrmot modelThis notebook demostractes how to use a model in eWatercycle by showing how to run a model for a particular catchment and a particular set of forcing data (ERA5). This notebook also contains the steps for the preprocessing of the forcing data into model specific datasets using ESMValTool.The model used in this notebook is: [MARRMoT](https://github.com/wknoben/MARRMoT) M01 (traditional bucket) (Knoben et al 2019, [10.5194/gmd-12-2463-2019](https://doi.org/10.5194/gmd-12-2463-2019)). MARRMoT is a suite of hydrological models and the chosen M01 model is the most simple model within it: it represents the entire catchment as a traditional single bucket. The catchment chosen as an example is the Merrimack. This model is too simple for the basin used as example and uncalibrated on top of that: this notebook is purely intended to demonstrate how to run a model and generate forcing data. It is not intended as an accurate simulations of discharge in this given basin, which is covered in the other, more complex, case study notebooks in this repository.Furthermore, this notebook shows that a model written in a language other than Python (in this case: Matlab) can still be run from a Jupyter notebook within eWaterCycle.  Import statementsWe'll be using the following modules<jupyter_code>import warnings
warnings.filterwarnings("ignore", category=UserWarning)

import logging
logger_esmvalcore = logging.getLogger('esmvalcore')
logger_esmvalcore.setLevel(logging.WARNING)

import pandas as pd
from cartopy.io import shapereader

import ewatercycle.forcing
import ewatercycle.models
import ewatercycle.analysis
import ewatercycle.observation.grdc<jupyter_output><empty_output><jupyter_text>Experiment settingsDischarge of the Merrimack river will be modelled for 1991. MARRMoT requires spinup, so 1990 is used to spin-up the model<jupyter_code>forcing_dataset_name = "ERA5"
experiment_start_date = "1990-01-01T00:00:00Z"
experiment_end_date = "1990-12-31T00:00:00Z"
shapefile = "settingFiles/Merrimack.shp"

# GRDC observation station for Merrimack river
grdc_station_id = "4147380"<jupyter_output><empty_output><jupyter_text>Model specific settingsThe maximum soil moisture storage is the only parameter in this simple model. Normally this parameter would be calibrated using a combination of forcing and observation data. Calibration of a model is shown in another notebook in this repository. In this notebook the focus is on showing how to run a model. Many models use different names for the same phenomena, in this case river discharge. Here we specify what the 'internal' name is for our variable of interest.<jupyter_code># Values based on previous calibration
experiment_maximum_soil_moisture_storage = 1999.0
experiment_initial_soil_moisture_storage = 0.9 * experiment_maximum_soil_moisture_storage

discharge_variable = "flux_out_Q"
# flux_out_Q unit conversion factor from mm/day to m3/s
conversion_mmday2m3s = 1 / (1000 * 86400)<jupyter_output><empty_output><jupyter_text>Pre processing model forcing using ESMValToolESMValTool (Righi et. al. 2020, [https://doi.org/10.5194/gmd-13-1179-2020](https://doi.org/10.5194/gmd-13-1179-2020)) is used to pre-process climate forcing data into model specific input data. See the article (link in readme file) for more details on ESMValTool. The cell below shows how the eWaterCycle platform does most of the heavy lifting and users only specify information relevant to their experiment.<jupyter_code>experiment_forcing = ewatercycle.forcing.generate(
    target_model='marrmot', 
    dataset=forcing_dataset_name, #ie. "ERA5"
    start_time=experiment_start_date,
    end_time=experiment_end_date, 
    shape=shapefile, #ie. the shape of the Merrimack basin
)<jupyter_output>{'auxiliary_data_dir': PosixPath('/mnt/data/climate-data/aux'),
 'compress_netcdf': False,
 'config_developer_file': None,
 'config_file': PosixPath('/home/rhut/.esmvaltool/config-user.yml'),
 'drs': {'CMIP5': 'default', 'CORDEX': 'default', 'OBS': 'default'},
 'exit_on_warning': False,
 'extra_facets_dir': (),
 'log_level': 'info',
 'max_parallel_tasks': None,
 'output_dir': PosixPath('/home/rhut/technicalPaperExampleNotebooks/esmvaltool_output/recipe_marrmot_20210930_125335'),
 'output_file_type': 'png',
 'plot_dir': PosixPath('/home/rhut/technicalPaperExampleNotebooks/esmvaltool_output/recipe_marrmot_20210930_125335/plots'),
 'preproc_dir': PosixPath('/home/rhut/technicalPaperExampleNotebooks/esmvaltool_output/recipe_marrmot_20210930_125335/preproc'),
 'profile_diagnostic': False,
 'remove_preproc_dir': False,
 'rootpath': {'OBS6': [PosixPath('/mnt/data/climate-data/obs6')],
              'default': [PosixPath('/mnt/data/climate-data')]},
 'run_dir': PosixPath('/home/rhut/technicalP[...]<jupyter_text>Forcing is written in a single Matlab file<jupyter_code>print(experiment_forcing)<jupyter_output>eWaterCycle forcing
-------------------
start_time=1990-01-01T00:00:00Z
end_time=1990-12-31T00:00:00Z
directory=/home/rhut/technicalPaperExampleNotebooks/esmvaltool_output/recipe_marrmot_20210930_125335/work/diagnostic_daily/script
shape=/home/rhut/technicalPaperExampleNotebooks/settingFiles/Merrimack.shp
forcing_file=marrmot_ERA5_Merrimack_1990_1990.mat<jupyter_text>Derived constantsMARRMoT works with expressing all hydrological fluxes in mm/s and stores in mm. This makes sense from the perspective of the model builders original purpose with MARRMoT: to easily compare different model structures on different regions. However, when comparing the predicted river discharge to observations measured in m3/s, the area of the basin is needed. This is derived from the shapefile in the cell below.<jupyter_code>shape = shapereader.Reader(experiment_forcing.shape)
record = next(shape.records())
merrimack_area = record.attributes["SUB_AREA"] * 1e6
print("The catchment area is:", merrimack_area)<jupyter_output>The catchment area is: 13016500000.0<jupyter_text>Setting up the model objectTo use a hydrological model, supported by ewatercycle, an object from its class must be instantiated. Information on the versions of models can be queried as shown below:<jupyter_code>ewatercycle.models.MarrmotM01.available_versions<jupyter_output><empty_output><jupyter_text>subsequently a model object is created:<jupyter_code>model = ewatercycle.models.MarrmotM01(version='2020.11', forcing=experiment_forcing)
print(model)<jupyter_output>eWaterCycle MarrmotM01
-------------------
Version = 2020.11
Parameter set = 
  None
Forcing = 
  eWaterCycle forcing
  -------------------
  start_time=1990-01-01T00:00:00Z
  end_time=1990-12-31T00:00:00Z
  directory=/home/rhut/technicalPaperExampleNotebooks/esmvaltool_output/recipe_marrmot_20210930_125335/work/diagnostic_daily/script
  shape=/home/rhut/technicalPaperExampleNotebooks/settingFiles/Merrimack.shp
  forcing_file=marrmot_ERA5_Merrimack_1990_1990.mat<jupyter_text>Create config file and start up container for Marrmot<jupyter_code>cfg_file, cfg_dir = model.setup(
    # No need to specifiy start and end date, using dates from forcing
    maximum_soil_moisture_storage=experiment_maximum_soil_moisture_storage,
    initial_soil_moisture_storage=experiment_initial_soil_moisture_storage,
)
print(cfg_file)
print(cfg_dir)
model.initialize(cfg_file)<jupyter_output><empty_output><jupyter_text>Run model & capture dischargeThis is the main loop of an eWaterCycle experiment. As long as the time variable of the model has not surpassed the end time of the experiment, `model.update()` runs a single timestep of the model. the `simulated_discharge` and `timestamps` arrays are filled with the calculated discharge and timestamps respectivly using the `model.get_value()` and `model.time_as_datetime.date()` functions.<jupyter_code>simulated_discharge = []
timestamps = []

while (model.time < model.end_time):
    model.update()
    timestamps.append(model.time_as_datetime.date())
    # Marrmot M01 is a lumped model, so only single value is returned
    value_in_mmday = model.get_value(discharge_variable)[0]
    # Convert from mm/day to m3/s
    value = value_in_mmday * merrimack_area * conversion_mmday2m3s
    simulated_discharge.append(value)<jupyter_output><empty_output><jupyter_text>The `timestamps` and `simulated_discharge` variables are now filled with datetime values and discharges for each simulated day.  Clean up after the model runThe models have to be 'finalized', which deletes any temporary files and the containers have to be shut down.<jupyter_code>model.finalize()<jupyter_output><empty_output><jupyter_text>Import GRDC observationsRead the GRDC data for the period of the experiment, for the station given above. If you are not on jupyter.ewatercycle.org or Cartesius, download the GRDC https://www.bafg.de/GRDC/EN/02_srvcs/21_tmsrs/riverdischarge_node.html and set the `grdc_location` field in `~/.config/.ewatercycle/ewatercycle.yaml` to your download location.<jupyter_code>observations, metadata = ewatercycle.observation.grdc.get_grdc_data(
    station_id=grdc_station_id,
    start_time=experiment_start_date,
    end_time=experiment_end_date,
)
observations<jupyter_output>GRDC station 4147380 is selected. The river name is: MERRIMACK RIVER.The coordinates are: (42.6459, -71.2984).The catchment area in km2 is: 12004.7. There are 0 missing values during 1990-01-01T00:00:00Z_1990-12-31T00:00:00Z at this station. See the metadata for more information.<jupyter_text>Plot the resultsCombine simulated and observated discharge into a single dataframe<jupyter_code>simulated_discharge_df = pd.DataFrame(
    {'simulation': simulated_discharge}, index=pd.to_datetime(timestamps)
)
observations_df = observations.rename(
    columns={'streamflow': 'observation'}
)
discharge = simulated_discharge_df.join(observations_df)
discharge
ewatercycle.analysis.hydrograph(
    discharge=discharge,
    reference='observation', 
    filename ='figures/case1_HelloWorldHydrograph.png'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08576515975863093, 'fraction_numerical': 0.038480561875556436, 'mean_word_length': 6.144876325088339, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 48, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '120051', 'license': 'Apache-2.0', 'n_tokens_mistral': 3486, 'n_tokens_neox': 3120, 'n_words': 892, 'path': 'Case1_Marrmot_Merrimack_ESMValTool.ipynb', 'repo_name': 'eWaterCycle/technicalPaperExampleNotebooks', 'size': '10109'}