<jupyter_start><jupyter_text>IntroIn this tutorial we will execute a `dvmdostem` simulation from a Jupyter Notebook (IPython Notebook). The tutorial assumes:* You have built all the `dvmdostem` Docker images (see the comments and directions in the [Dockerfile](https://github.com/ua-snap/dvm-dos-tem/blob/master/Dockerfile) and [Dockerfile-mapping-support](https://github.com/ua-snap/dvm-dos-tem/blob/master/Dockerfile-mapping-support). * You can launch the stack using `docker compose` (see directions and examples in the [docker-compose.yml](https://github.com/ua-snap/dvm-dos-tem/blob/master/docker-compose.yml) file).* You have the volumes setup as specified in the project's `docker-compose.yml` file and you can access the files both from your host computer and from within the docker container(s).* You have installed the Jupyter package and can run jupyter notebooks, specifically an `IPython` notebook.> Note about packages and environments: the demo here shows a mixture of running dvm-dos-tem's supporting Python scripts "natively" (i.e. on your host computer, not in a Docker container) and running dvm-dos-tem's supporting Python scripts inside the Docker container. For the most consistent Python environment, it is best to run everyting through (inside) the Docker containers. However sometimes the extra typing required is onerous (`docker compose exec dvmdostem-run ...`), and there is a little extra overhead involved in interacting with the containers. So if you have the appropriate packages installed on your host machine you can run some of the scripts on your host as it shown here. This takes a little care to keep the paths straight for each command, i.e. whether you are referncing the path inside the guest/container, or on your host machine.> Note that while `dmvdostem`'s supporting scripts are largely written in Python, for the most part there is not an exposed Python API. Instead the scripts are generally provided with a command line interface. So much of this tutorial, while running inside an IPython notebook, could easily be run directly from your shell.For convenience we will add the `dvm-dos-tem` repo directory to our `PATH` on the host **before** launching the notebook (this should already be done for you inside the Docker container):```bash$ cd ~/path/to/your/dvm-dos-tem$ export PATH=$(PWD)/scripts:$(PWD):$PATH```Now start the notebook server:```bash$ jupyter notebook```and open this notebook in a browser tab. **Note that when you open the notebook, the current working directory of the notebook will be the dvm-dos-tem scripts directory. In order for the docker commands to work correctly, you must change your working directory for this notebook to the main dvm-dos-tem repo directory, which contains the Dockerfile(s).** You can do this a variety of ways, but the easiest is the builtin IPython magic, `%cd `. There is also the useful `%pwd` function to see where you are.  SetupAs noted in [this wiki page](https://github.com/ua-snap/dvm-dos-tem/wiki/How-To:-Run-dvmdostem-and-plot-output-via-Docker) there are 6 general steps for any modeling task: 1. Decide where on your computer you want to store your model run(s).2. Decide what spatial (geographic) area you want to run.3. Decide what variables you want to have output4. Decide on all other run settings/parameters: * Which stages to run and for how many years. * Is the community type (CMT) fixed or driven by input vegetation.nc map? * For which stages should the output files be generated and saved? * Calibration settings if necessary (`--cal-mode`). * Any other command line options or environment settings.5. Launch the run6. Make plots or other analysis. Setup data location and geographic location.Start by setting up a directory for this example run. There are several ways you could do this. For this example we will run the script inside the docker container. Hence the paths will be from the point of view of inside the docker container.<jupyter_code># Cleanup:
!docker compose exec dvmdostem-run rm -r /data/workflows/testcase_0001
!docker compose exec dvmdostem-run scripts/setup_working_directory.py \
--input-data-path /data/input-catalog/cru-ts40_ar5_rcp85_ncar-ccsm4_TOOLIK_FIELD_STATION_10x10/ \
/data/workflows/testcase_0001<jupyter_output>Namespace(copy_inputs=False, input_data_path='/data/input-catalog/cru-ts40_ar5_rcp85_ncar-ccsm4_TOOLIK_FIELD_STATION_10x10/', new_directory='/data/workflows/testcase_0001', no_cal_targets=False)<jupyter_text>Now note that if you investigate **from your host** (i.e. not inside the docker container) you can see the new directory you just created (in my case, I keep the workflows up two directories from my dvm-dos-tem repo; your paths might be different):<jupyter_code>!ls ../dvmdostem-workflows/testcase_0001/<jupyter_output>[1m[36mcalibration[m[m [1m[36mconfig[m[m      [1m[36moutput[m[m      [1m[36mparameters[m[m  run-mask.nc<jupyter_text>Notice that you can see the same folders and files by running `ls` from within the docker container. Notice the paths are different because you are referencing the locations **inside** the container as oppsed to from your host computer:<jupyter_code>!docker compose exec dvmdostem-run ls /data/workflows/testcase_0001<jupyter_output>calibration  config  output  parameters  run-mask.nc<jupyter_text>Adjust spatial maskNow adjust your run mask so as to only run one pixel:<jupyter_code>!docker compose exec dvmdostem-run runmask-util.py --reset \
--yx 0 0 \
--show \
/data/workflows/testcase_0001/run-mask.nc<jupyter_output>========== BEFORE ==================================

** Keep in mind that in this display the origin is the upper 

** left of the grid! This is opposite of the way that ncdump 

** and ncview display data (origin is lower left)!!



'/data/workflows/testcase_0001/run-mask.nc'

<class 'netCDF4._netCDF4.Variable'>

int64 run(Y, X)

    grid_mapping: albers_conical_equal_area

unlimited dimensions: 

current shape = (10, 10)

filling on, default _FillValue of -9223372036854775806 used

[[1 0 0 0 0 0 0 0 0 0]

 [0 1 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]

 [0 0 0 0 0 0 0 0 0 0]]



Setting all pixels in runmask to '0' (OFF).

Turning pixel(y,x) (0,0) to '1', (ON).

========== AFTER ==================================

** Keep in mind that in this display the origin is the upper 

** left of the grid! This is opposite o[...]<jupyter_text>Choose output variables OK, now its time to turn on some outputs. In this case, lets keep it simple and just turn on monthly GPP for PFTs. The output from this command needs a wider display for easier reading. First check on what is already on:<jupyter_code>!docker compose exec dvmdostem-run outspec_utils.py -s /data/workflows/testcase_0001/config/output_spec.csv<jupyter_output>Name                Units       Yearly      Monthly        Daily          PFT Compartments       Layers    Data Type     Description

                 GPP            g/m2/time            y                   invalid                                invalid       double     GPP<jupyter_text>This is super annoying because it needs a wider screen to display this table nicely. But we can use `pandas` to display nicely in this notebook.<jupyter_code>import pandas as pd
outspec = pd.read_csv('../dvmdostem-workflows/testcase_0001/config/output_spec.csv')
outspec.head(15)<jupyter_output>/Users/tobeycarman/.pyenv/versions/3.8.2/envs/py38-ddt2/lib/python3.8/site-packages/pandas/compat/__init__.py:120: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)<jupyter_text>Now use the utility helper script to change the file:<jupyter_code>!docker compose exec dvmdostem-run outspec_utils.py \
/data/workflows/testcase_0001/config/output_spec.csv \
--on GPP p m

!docker compose exec dvmdostem-run outspec_utils.py \
/data/workflows/testcase_0001/config/output_spec.csv \
--on CMTNUM y<jupyter_output>Name                Units       Yearly      Monthly        Daily          PFT Compartments       Layers    Data Type     Description
                 GPP            g/m2/time            y            m      invalid            p                   invalid       double     GPP
                Name                Units       Yearly      Monthly        Daily          PFT Compartments       Layers    Data Type     Description
              CMTNUM                    m            y      invalid      invalid      invalid      invalid      invalid          int     Community Type Number<jupyter_text>Adjust other settingsNow modify the config file to turn on equilibrium stage outputs. Again there are about 100 ways to do this (on the host, in the container, with a text editor, programatically, etc). For these purposes, we will work from the host, mostly so we can record (and run) the Python snippet here in the notebook:<jupyter_code>import json

CONFIG_FILE = '../dvmdostem-workflows/testcase_0001/config/config.js'

# Read the existing data into memory
with open(CONFIG_FILE, 'r') as f:
    config = json.load(f)

# Modify it
config['IO']['output_nc_eq'] = 1

# Write it back..
with open(CONFIG_FILE, 'w') as f:
    json.dump(config, f, indent=2)<jupyter_output><empty_output><jupyter_text>Note you can check on the file with `!cat {CONFIG_FILE}`.  RUNNow finally we will launch our `dvmdostem` run. For these purposes, we are just doing a 200 year equlibrium run. We are going to force this to run as CMT 4 (Tussock Tundra).<jupyter_code>!docker compose exec --workdir /data/workflows/testcase_0001 dvmdostem-run \
dvmdostem -p 50 -e 200 -s 0 -t 0 -n 0 -l err --force-cmt 4<jupyter_output>Setting up logging...
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 0
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 1
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 2
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 3
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 4
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 5
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 6
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 7
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 8
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 9
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 10
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 11
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 12
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 13
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 14
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 15
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 16
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 17
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 18
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 19
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 20
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 21
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 22
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 23
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 24
[err] [PRE-RUN->Y] y: 0 x: 0 Year: 25
[...]<jupyter_text>Thats it! If we look in the output directory, we expect to see one output file for the equlibrium stage, GPP:<jupyter_code>!docker compose exec dvmdostem-run ls /data/workflows/testcase_0001/output
!ncdump ../dvmdostem-workflows/testcase_0001/output/run_status.nc<jupyter_output>netcdf run_status {
dimensions:
	Y = 10 ;
	X = 10 ;
variables:
	int run_status(Y, X) ;
data:

 run_status =
  100, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ;
}<jupyter_text>PlottingAgain, this can be done a few ways (host or guest). For these purposes, we will run on the host.<jupyter_code>import netCDF4 as nc
import scripts.output_utils as ou
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>The output_utils script has a bunch of unfinished plotting tools in it and some useful funcitons for summarizing outputs (i.e. summing PFTs or converting monthly outputs to yearly).First just see what output files we have to work with:<jupyter_code>!ls ../dvmdostem-workflows/testcase_0001/output/<jupyter_output>CMTNUM_yearly_eq.nc restart-eq.nc       restart-sc.nc       restart-tr.nc
GPP_monthly_eq.nc   restart-pr.nc       restart-sp.nc       run_status.nc<jupyter_text>Summarize to yearly, plot first 5 PFTs<jupyter_code>ds = nc.Dataset("../dvmdostem-workflows/testcase_0001/output/GPP_monthly_eq.nc")
gpp = ds.variables['GPP'][:]
yrly_gpp = ou.sum_monthly_flux_to_yearly(gpp)
for pft in range(0,5):
    plt.plot(yrly_gpp[:,pft,0,0], label='pft{}'.format(pft))
plt.xlabel('run year')
plt.ylabel('{} ({})'.format('GPP', 'g/m2/year'))
_ = plt.legend()<jupyter_output><empty_output><jupyter_text>Summarize to yearly, sum PFTs<jupyter_code>plt.plot(ou.sum_across_pfts(yrly_gpp)[:,0,0], label='all pfts')
plt.xlabel('run year')
_ = plt.ylabel('GPP (g/m2/year)')<jupyter_output><empty_output><jupyter_text>Plot the last 10 years at monthly resolution, first 5 PFTs<jupyter_code>import netCDF4 as nc
ds = nc.Dataset("../dvmdostem-workflows/testcase_0001/output/GPP_monthly_eq.nc")
gpp = ds.variables['GPP'][:]
for pft in range(0,5):
    plt.plot(gpp[-120:,pft,0,0], label='pft{}'.format(pft))
plt.xlabel('month')
plt.ylabel('GPP ({})'.format(ds.variables['GPP'].units))
_ = plt.legend()<jupyter_output><empty_output><jupyter_text>Unfinished output_utils.py plot function<jupyter_code>ou.plot_basic_timeseries(['GPP'],0,0,'monthly',['eq'],'../dvmdostem-workflows/testcase_0001/output/')<jupyter_output>Trying to open:  ../dvmdostem-workflows/testcase_0001/output/GPP_monthly_eq.nc
(2400, 10, 10, 10)
{'fraction_non_alphanumeric': 0.1131440281030445, 'fraction_numerical': 0.03790983606557377, 'mean_word_length': 3.7185773480662982, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 83, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '284469', 'license': 'MIT', 'n_tokens_mistral': 5133, 'n_tokens_neox': 4582, 'n_words': 1720, 'path': 'scripts/docker-ipython-notebook-run.ipynb', 'repo_name': 'valeriabriones/dvm-dos-tem', 'size': '13664'}