<jupyter_start><jupyter_text>Overview Tasmania example of the AWRA-CMS functionalityThe purpose of this notebook is to give you an overview of the AWRA-CMS functionality for a given example.  We have previously given a description of the AWRA-L model in python in the notebook: [1.3_AWRA-L_deconstructed_-_Python_version_of_AWRA-L][1.3_AWRA-L_deconstructed_-_Python_version_of_AWRA-L]: /tree/Training/Basics/1.3_AWRA-L_deconstructed_-_Python_version_of_AWRA-L.ipynb. A quick refresher of the components of the AWRA MS:- **Simulation** *Module:* **awrams.simulation**  A specified run of the AWRA model over a time period and spatial extent- **Calibration** *Module:* **awrams.calibration**  Using an optimisation algorithm and an objective function to obtain the optimum set of AWRA-L model parameters for a given time period, extent, etc.- **Visualisation** *Module:* **awrams.visualisation** A figure showing AWRA-L inputs and outputs as maps and time-series- **Benchmarking** *Module:* **awrams.benchmarking** Validating model outputs compared to observed data or other model simulations- **Extraction**  Extraction of point or catchment average data from the AWRA-L grids- **Utilities** *Module:* **awrams.utils**  Functionality that is required in more than one of the AWRA MS components Let's also define some key terms:- **Nodegraph**     - Node - a single input component     - Graph - the order that these input components are defined    - In the AWRA MS we have an 'input nodegraph' and an 'output nodegraph'- **Forcing data**      Gridded inputs to our model (i.e. precipitation, solar exposure, max temperature, min temperature- **Climatology**     Mean climate state for each day/month of the year used by the AWRA MS to infill gaps in the forcing data- **Initial states**     The AWRA-L model has a number of states that change from one time-step to the next (i.e. mleaf, s0, ss, sd, sg, sr). Without specifying these the default settings assume all soil and ground stores are half-full.- **Spatial extent**  This is the area that we will undertake a calibration or simulation or visualisation    For more definition of the AWRA-CMS see the user guide: [AWRA-CMS User guide ][AWRA-CMS User guide ]: /tree/Training/AWRA_Landscape_Community_Modelling_System_User_Guide_2017.pdf **Tasmania example:**Rather than using a simulation from the whole continent, we would like to undertake a model calibration using a set of five Tasmanian catchments only (IDs: 306119, 308145, 312061, 318150, 318852). Once we have calibrated the model we will simulate over Tasmania and visualise these results.The notebook follows a typical workflow of calibration, simulation, benchmarking and visualisation with the following steps: 1. Import required libraries2. Configure model3. Calibrate the AWRA-L model to the Tasmanian catchments4. Simulation over Tasmania5. Benchmark to standard AWRA-L calibration6. Visualisation of results*After this example overview session we will go through all of these steps in much greater detail during the upcoming sessions.* ***  1. Import required libraries<jupyter_code># Get os path
from os.path import join

# import various python modules
import pandas as pd
import h5py
from matplotlib import pyplot as plt

# Import AWRA MS utilities
from awrams.utils import config_manager, extents, gis
from awrams.utils import datetools as dt
from awrams.utils.gis import ShapefileDB
from awrams.utils.nodegraph import nodes, graph

# Import AWRA MS calibration
from awrams.calibration import cluster, support
from awrams.calibration.launch_calibration import run_from_pickle
from awrams.calibration.objectives import test_objectives as tobj
from awrams.calibration.optimizers import sce
# Get the base data path from the system profile
sys_profile = config_manager.get_system_profile()
sys_settings = sys_profile.get_settings()

base_data_path = sys_settings['DATA_PATHS']['BASE_DATA']<jupyter_output><empty_output><jupyter_text>Questions?  2. Configure model<jupyter_code># Instantiate the AWRA-L model with default model settings
model_profile = config_manager.get_model_profile('awral', 'v6_default')
awral = model_profile.get_model()
#Specify running and calibration period
run_period = dt.dates('2009 - 2011')
eval_period = dt.dates('2010 - 2011')<jupyter_output><empty_output><jupyter_text>2.1 Specify calibration catchments<jupyter_code># Set list of catchment ids
id_list = ['306119', '308145', '312061', '318150', '318852']
# Get the default extent of the calibration (This is the whole of Australia capturing all catchments)
catchments_shapefile = join(base_data_path, 'spatial/shapefiles/Final_list_all_attributes.shp')
calvalshapefile = ShapefileDB(catchments_shapefile)
def_extent = extents.get_default_extent() 

## Create a dict with multiple Tasmanian catchment extents
cal_dict = {}
for catchment_id in id_list:
    cal_dict[catchment_id] = calvalshapefile.get_extent_by_field(
            'StationID', catchment_id.zfill(6), parent_extent=def_extent)

cal_dict
# Visualise the Tasmanian catchment locations (Note that some of these are quite small to see)
import awrams.visualisation.results as res
import awrams.visualisation.vis as vis

vis.show_extent(cal_dict['312061'],def_extent)
vis.show_extent(cal_dict['318150'],def_extent)
vis.show_extent(cal_dict['306119'],def_extent)
vis.show_extent(cal_dict['318852'],def_extent)
vis.show_extent(cal_dict['308145'],def_extent)
# Specify the location of the observed data
observations = dict(qtot=base_data_path + '/observations/runoff/awrams_v5_cal_qobs.csv')<jupyter_output><empty_output><jupyter_text>2.2 Specify input mapping (input nodegraph)<jupyter_code>input_map = model_profile.get_input_mapping()
def insert_solar_climatology(imap):
    from awrams.utils.nodegraph import nodes

    climatologies = {
        'solar': (
            base_data_path + '/training/simulation/climatology/Rad_1990_2009.nc', 'solar_exposure_day'
        )
    }

    imap = imap.copy()

    imap['solar_f_orig'] = imap['solar_f'] #'Move' the forcing node to a new name
    imap['solar_climatology_f'] = nodes.monthly_climatology(*climatologies['solar']) # Loads monthly climatology from default file

    # Replace 'solar_f' with infilled data
    # This ensures that any other nodes in the graph who use solar_f as input will automatically receive the infilled data
    # nodes.gap_filler takes the first argument as 'gappy' data, and in infills with data from the second argument
    imap['solar_f'] = nodes.gap_filler('solar_f_orig', 'solar_climatology_f')
    
    return imap
input_map = insert_solar_climatology(input_map)
input_map<jupyter_output><empty_output><jupyter_text>Questions?  3. Calibrate the AWRA-L model to the Tasmanian catchments<jupyter_code>#Select optimisation algorithm
evolver_spec = support.EvolverSpec(sce.CCEvolver,
                                   evolver_run_args=dict(n_offspring=1, n_evolutions=5, elitism=2.0))

# Please note that these specifications are designed to reduce runtime for demo purposes
optimizer_spec = support.OptimizerSpec(sce.ShuffledOptimizer,
                                       evolver_spec=evolver_spec,
                                       n_complexes=3, # (Initial) number of complexes
                                       max_nsni=300, # Maximum number of shuffles without improvement
                                       min_imp = 0.1, # Minimum proportion of improvement to occure over max_nsni
                                       min_complexes=1, # Mininum number of complexes
                                       max_eval=1000) # Maximum model evaluations before terminating

#optimizer_spec = OptimizerSpec(SobolOptimizer,threshold = 0.005,max_eval =10000)
#Import objective function
local_objfspec = support.ObjectiveFunctionSpec(tobj.TestLocalSingle)  # this function to load up the ObjFunc comes from awrams.calibration.support
global_objfspec = tobj.TestGlobalSingle
objective_spec = support.ObjectiveSpec(global_objfspec, local_objfspec, observations, eval_period)
# Build spec dict
# Get the input mapping and model
node_mapping = input_map
model = awral#callable_to_funcspec(awral)
# Create the calibration specification dictionary

'''
User specifiable calibration description
'''
cal_spec = {}
cal_spec['optimizer_spec'] = optimizer_spec
cal_spec['objective_spec'] = objective_spec
cal_spec['extent_map'] = cal_dict
cal_spec['run_period'] = run_period
cal_spec['eval_period'] = eval_period
cal_spec['model'] = model
cal_spec['node_mapping'] = node_mapping
#cal_spec['logfile'] = '/short/er4/dss548/calibration_res.h5'
cal_spec['logfile'] = './tas_cal.h5'
# Save a file containing all the calibration specifications
nnodes = 1
ncores = 4
cluster.build_pickle_from_spec(cal_spec, ncores, nnodes, 'tas_cal.pkl')
print("Depending on the specifications of your machine the calibration will run for some time")
cal = run_from_pickle('./tas_cal.pkl') # run the calibration
# Obtain the best (minimum score) parameter set
cr = support.CalibrationResults('./tas_cal.h5')
best_params = cr.get_best_paramset()
new_params = pd.DataFrame(best_params, columns=['value'])
new_params['MemberName'] = new_params.index
# Run the model with new parameter set. Change the parameter set in the input map
def change_model_parameters(imap, param_df):
    """
    the intent of this function is to replace the values in imap (your current configuration) 
    by values in a new parameter dataframe (param_df)
    """
    
    import pandas as pd
    
    variable_names = param_df['MemberName']
    variable_value = param_df['value']
      
    for k,v in zip(variable_names, variable_value):
        imap[k.lower()].args['value'] = v
        
change_model_parameters(input_map, new_params)<jupyter_output><empty_output><jupyter_text>Questions?  4. Simulate over Tasmania<jupyter_code>#Load up a  server simulator to run the model to write outputs out
from awrams.simulation import server
# Confirm output settings
omap = awral.get_output_mapping()

def build_output_mapping(output_map, outpath):
    from awrams.utils.nodegraph import nodes
   
    FILE_OUTPUT_VARS = ['s0', 'ss', 'sd', 'qtot', 'etot']
    
    for f in FILE_OUTPUT_VARS:
            output_map[f + '_ncsave'] = nodes.write_to_annual_ncfile(outpath, f)

    return output_map
# Set simulation output path
outpath = './_results_tas_cal/'
omap = build_output_mapping(omap, outpath)
from awrams.utils import datetools as dt
period = dt.dates('2010 - 2011')
## The functionality to model multiple extents at a time is coming. In the meantime, either:
## model 1 catchment at a time or model the entire bounding box..ie. Tasmania coords[-39.5:-44,143.5:149]
sim = server.SimulationServer(awral)
sim.run(input_map, omap, period, def_extent.icoords[-39.5:-44, 143.5:149])
# You may get a netCDF error, if so you may need to delete the previous simulation "./_results_tas_cal/" and then try again
# Extract outputs into csv's for each catchment
import os
import pandas as pd
from awrams.utils import extents
from awrams.utils.gis import ShapefileDB
from awrams.utils.io.data_mapping import SplitFileManager
from awrams.utils.processing.extract import extract_from_filemanager

catchments_shapefile = join(base_data_path, 'spatial/shapefiles/Final_list_all_attributes.shp')
catchments = ShapefileDB(catchments_shapefile)

var_name = 'qtot'
model_data_path = os.getcwd() + '/_results_tas_cal/'
period = dt.dates('2010 - 2011')

pattern = model_data_path + '/%s*' % var_name
sfm = SplitFileManager.open_existing(model_data_path, pattern, var_name)
georef = sfm.get_extent()
    
df = extract_from_filemanager(sfm, cal_dict, period)
df
df.plot()
df.to_csv('./tas_cal_qtot.csv')<jupyter_output><empty_output><jupyter_text>Questions?  5. Benchmark to observed data and a standard AWRA-L calibration<jupyter_code># Import Benchmarking module
from awrams.benchmarking.benchmark import Benchmark
obs_csv = base_data_path + '/observations/runoff/awrams_v5_cal_qobs.csv'
# Define Benchmarking object
q = Benchmark("QObs", "runoff")

# Specify benchmarking period
q.period = dt.dates("2010 -2011")
# Add observations and catchment subset [the id list needs to be present in the column names of the observation file]
q.load(obs_csv, id_list=id_list)
## Add model outputs
## Default AWRAL calibration
csv_data = base_data_path + '/training/benchmarking/runoff/AWRAMSI_v5QES_AWRAL_qtot_avg.csv'
q.add_model("AWRAMSI_v5_0_AWRAL", data_csv=csv_data)

## Regional Tasmanian calibration
csv_data = './tas_cal_qtot.csv'
q.add_model("Tas_Cal", data_csv=csv_data)
# View benchmarking statistics from both the standard AWRA calibration (AWRAMSI_v5_0_AWRAL) 
# and our Tasmanian calibration (Tas_Cal)
q.benchmark.stat_percentiles('nse', freq='d')
q.benchmark.stat(statistic='nse')
q.benchmark.stat() 
# plot a timeseries of observed (black) and predicted streamflow from both the standard 
# AWRA calibration (AWRAMSI_v5_0_AWRAL) and our Tasmanian calibration (Tas_Cal)
p = q.benchmark.plot_timeseries('306119', ylim=[0,50], freq = 'd')<jupyter_output><empty_output><jupyter_text>Questions?  6. Visualise over the spatial extent (Tasmania)<jupyter_code># Import AWRA MS modules
import awrams.visualisation.vis as vis
import awrams.visualisation.results as res
import awrams.utils.extents as extents
# Load the results from the simulation
results = res.load_results('./_results_tas_cal')
# Here we visualise all of the outputs, on a single day, over the entire spatial extent of the simulation
results[:,'march 2010',:].spatial()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08444772811230145, 'fraction_numerical': 0.018396749168821575, 'mean_word_length': 5.029398663697105, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '659007', 'license': 'NetCDF', 'n_tokens_mistral': 4465, 'n_tokens_neox': 4116, 'n_words': 1292, 'path': 'Training/Overview/Overview_of_the_AWRA_CMS_functionality.ipynb', 'repo_name': 'kaamilah/awra_cms', 'size': '13535'}