<jupyter_start><jupyter_text>Introduction(NOTE: This notebook is intended for use with the slides found [here](https://github.com/cropsinsilico/CiS2021-hackathon/blob/main/slides.pdf)).This is a Jupyter notebook. It allows us to run code (in this case Python) alongside text in different "cells". This cell is a markdown cell that can display text and html, the next cell is a code cell.In the code cells (prefixed by `In [ ]:`), you can assign variables, perform calculations or call external functions/classes. You can run code cells by selecting the cell (so that a blue or green box appears around it) and then clicking the run button (located at the top of the page) or pressing `Shift+Enter` together. Then a number will appear inside the brackets indicating the order of when the cell was executed. Output from the cell will be displayed below it with the `Out[]:` prefix where the number in the brackets indicates the input cell that generated it.<jupyter_code>x = 1
y = 3
z = (x + y)**3
z<jupyter_output><empty_output><jupyter_text>Any Python code can be used, and we can import external packages as well just like in Python scripts. Cells can also use any variables created in any previously executed cell. The cell below imports some tools that will be used in the rest of this notebook.<jupyter_code>from yggdrasil import tools  # Displaying syntax highlighted source code
from yggdrasil.runner import run  # Running integrations
import trimesh  # Load & display 3D meshes<jupyter_output><empty_output><jupyter_text>The notebook can also display plots, 3D graphics, and interactive widgets. The cell below uses `trimesh` to load and display a 3D mesh. You can drag the image to rotate the object and zoom in/out by scrolling over the image. This type of display will be used in some the examples today to display output.<jupyter_code>fname = 'meshes/plants-2.obj'
mesh = trimesh.load_mesh(fname)
mesh.show()<jupyter_output><empty_output><jupyter_text>Integrating Models as Functionsyggdrasil provides interfaces in several languages that can be used to open connections with other models, but in many cases, making a model work in integrations can be done by allowing yggdrasil to wrap a function that executes the model calculations.For example, the model displayed by the cell below calculates (albeit poorly) the intensity of light for a given day of the year and height from the ground. It is written as a Python function that takes `doy` (day of the year) and `height` as inputs and returns the intensity as output. ![image-2.png](attachment:image-2.png)<jupyter_code>tools.display_source('models/light_v0.py', number_lines=True)<jupyter_output><empty_output><jupyter_text>To use this model with yggdrasil, no modification of the code is necessary. The only new material required is a YAML configuration file. The cell below displays the YAML for the model above.<jupyter_code>tools.display_source('yamls/light_v0_python.yml', number_lines=True)<jupyter_output><empty_output><jupyter_text>This alone will not allow yggdrasil to run as the model inputs & output are not connected to anything. The YAML file below declares the connections that should be made to get input from the tab-delimited table `input/light_v0.txt` (also shown below) and direct output to the file `output/light_v0.txt`.Since the input/output to the light model is not explicitly defined in the model YAML above, yggdrasil assumes that all inputs will come from the same channel named `:input` and names the output channel `:output`.![image-2.png](attachment:image-2.png)<jupyter_code>tools.display_source('yamls/connections_v0.yml', number_lines=True)<jupyter_output><empty_output><jupyter_text>To run the light-to-file integration defined in these two YAML files, the `run` function is called with the paths to the YAML files as input. This is equivalent to calling `yggrun yamls/light_v0_python.yml yamls/connections_v0.yml --production-run` from the command line.The output from the yggdrasil integration will include output from the models themselves, some information about what stage each model is in, and the duration of different stages in the integration take to complete. In the case of running multiple models, the output from different models will often be interwoven and does not necessary indicate the order that models are executed in.<jupyter_code>run(['yamls/light_v0_python.yml', 'yamls/connections_v0.yml'], production_run=True)<jupyter_output><empty_output><jupyter_text>The cell below will display the contents of the output file `output/light_v0.txt` following the run.<jupyter_code>tools.display_source('output/light_v0.txt', number_lines=True)<jupyter_output><empty_output><jupyter_text>Command Line InterfaceThe examples above have been using yggdrasil's Python interface to run the integration, but that is not necessary. yggdrasil has a command line utility for running integration `yggrun` which takes YAML paths as inputs. The `production_run` KeywordYou may have noticed that we passed the `production_run` keyword to the `run` API function with a value of `True`. When set to `True`, yggdrasil turns of several safe guards that increase run-time. These include things like checking data formats and validating inputs/outputs to/from framework components. It is highly recommended, that `production_run` is only set to `True` when you are done testing an integration and are ready for a "production run" that requires higher performance. The `production_run` flag can also be passed to the command line interface `yggrun` as `--production-run`. Similarly in Other LanguagesNOTE: Units must be explicitly added via a `datatype` entry in the model yaml for the compiled languages (i.e. C, C++, & Fortran) C++ Version<jupyter_code>tools.display_source('models/light_v0.cpp', number_lines=True)
tools.display_source('yamls/light_v0_cpp.yml', number_lines=True)
run(['yamls/light_v0_cpp.yml', 'yamls/connections_v0.yml'], production_run=True)
tools.display_source('output/light_v0.txt', number_lines=True)<jupyter_output><empty_output><jupyter_text>Fortran Version<jupyter_code>tools.display_source('models/light_v0.f90', number_lines=True)
tools.display_source('yamls/light_v0_fortran.yml', number_lines=True)
run(['yamls/light_v0_fortran.yml', 'yamls/connections_v0.yml'], production_run=True)
tools.display_source('output/light_v0.txt', number_lines=True)<jupyter_output><empty_output><jupyter_text>R Version<jupyter_code>tools.display_source('models/light_v0.R', number_lines=True)
tools.display_source('yamls/light_v0_R.yml', number_lines=True)
run(['yamls/light_v0_R.yml', 'yamls/connections_v0.yml'], production_run=True)
tools.display_source('output/light_v0.txt', number_lines=True)<jupyter_output><empty_output><jupyter_text>Integrating Models via InterfaceThe function wrapping method of yggdrasil works in many cases, but not all. When a model must send or receive data to/from another model mid-calculation or the model algorithm is written such that writing it as a function would be unwieldy, the yggdrasil interface can be used directly.For example, the model below simulates growth of a 3D shoot structure over time and is executed via the command line with parameters controlling the how long the simulations runs and what the initial mesh looks like.<jupyter_code>tools.display_source('models/shoot_v0.py', number_lines=True)<jupyter_output><empty_output><jupyter_text>![image-2.png](attachment:image-2.png)We can run this model via the command line or via yggdrasil using the YAML displayed below which runs the model for 48 hrs with a time step of 6 hrs and does not handle any input or output.<jupyter_code>tools.display_source('yamls/shoot_v0.yml', number_lines=True)
run('yamls/shoot_v0.yml', production_run=True)<jupyter_output><empty_output><jupyter_text>The final mesh from this simulation is displayed by the cell below using the `trimesh` package.<jupyter_code>mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.show()<jupyter_output><empty_output><jupyter_text>If we want to determine the light intensity at the top of the plant at each timestep, re-writing this model as a function and allowing yggdrasil to wrap it requres a lot of modification to the original code. Instead we can create an output channel via the yggdrasil Python interface with minimal modification to the code.The cell below shows the diff for an updated version of this model that does this.1. Checks if the yggdrasil version of the code should be run1. Imports the relevant yggdrasil modules and functions and opens an output channel with the name `height`1. Sends the time and maximum height of the mesh to the `height` output channel with units<jupyter_code>tools.display_source_diff('models/shoot_v0.py', 'models/shoot_v1.py', number_lines=True)<jupyter_output><empty_output><jupyter_text>The above model can be run in the exact same manner as the original without yggdrasil. The YAML diff displayed in the cell below shows the changes necessary to connect the `height` output to a table file `output/height.txt` in the absence of any other connection.<jupyter_code>tools.display_source_diff('yamls/shoot_v0.yml', 'yamls/shoot_v1.yml', number_lines=True)<jupyter_output><empty_output><jupyter_text>![image-2.png](attachment:image-2.png)The cell below runs the 1-model integration defined in the YAML above.<jupyter_code>run(['yamls/shoot_v1.yml'], production_run=True)<jupyter_output><empty_output><jupyter_text>The resulting mesh and `output/height.txt` file is displayed by the next two cells.<jupyter_code>mesh = trimesh.load_mesh('output/mesh_008.obj')
mesh.show()
tools.display_source('output/height.txt')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07337265859463935, 'fraction_numerical': 0.007140639552933871, 'mean_word_length': 6.228122662677636, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '665725', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2860, 'n_tokens_neox': 2750, 'n_words': 1114, 'path': '00-intro.ipynb', 'repo_name': 'langmm/CiS2021-hackathon', 'size': '9663'}