<jupyter_start><jupyter_text>Introduction to NeuralHydrology**Before we start**- This tutorial is rendered from a Jupyter notebook that is hosted on GitHub. If you want to run the code yourself, you can find the notebook and configuration files [here](https://github.com/neuralhydrology/neuralhydrology/tree/master/examples/01-Introduction).- To be able to run this notebook locally, you need to download the publicly available CAMELS US rainfall-runoff dataset. See the [Data Prerequisites Tutorial](data-prerequisites.nblink) for a detailed description on where to download the data and how to structure your local dataset folder.The Python package NeuralHydrology was was developed with a strong focus on research. The main application area is hydrology, however, in principle the code can be used with any data. To allow fast iteration of research ideas, we tried to develop the package as modular as possible so that new models, new data sets, new loss functions, new regularizations, new metrics etc. can be integrated with minor effort.There are two different ways to use this package:1. From the terminal, making use of some high-level entry points (such as `nh-run` and `nh-run-scheduler`)2. From any other Python file or Jupyter Notebook, using NeuralHydrology's APIIn this tutorial, we will give a very short overview of the two different modes.Both approaches require a **configuration file**. These are `.yml` files which define the entire run configuration (such as data set, basins, data periods, model specifications, etc.). A full list of config arguments is listed in the [documentation](https://neuralhydrology.readthedocs.io/en/latest/usage/config.html) and we highly recommend to check this page and read the documentation carefully. There is a lot that you can do with this Python package and we can't cover everything in tutorials.For every run that you start, a new folder will be created. This folder is used to store the model and optimizer checkpoints, train data means/stds (needed for scaling during inference), tensorboard log file (can be used to monitor and compare training runs visually), validation results (optionally) and training progress figures (optionally, e.g., model predictions and observations for _n_ random basins). During inference, the evaluation results will also be stored in this directory (e.g., test period results). TensorBoard loggingBy default, the training progress is logged in TensorBoard files (add `log_tensorboard: False` to the config to disable TensorBoard logging). If you installed a Python environment from one of our environment files, you have TensorBoard already installed. If not, you can install TensorBoard with:```pip install tensorboard``` To start the TensorBoard dashboard, run:```tensorboard --logdir /path/to/run-dir```You can also visualize multiple runs at once if you point the `--logdir` to the parent directory (useful for model intercomparison) File loggingIn addition to TensorBoard, you will always find a file called `output.log` in the run directory. This file is a dump of the console output you see during training and evaluation. Using NeuralHydrology from the Terminal nh-runGiven a run configuration file, you can use the bash command `nh-run` to train/evaluate a model. To train a model, use```bashnh-run train --config-file path/to/config.yml```to evaluate the model after training, use```bashnh-run evaluate --run-dir path/to/run-directory``` nh-run-schedulerIf you want to train/evaluate multiple models on different GPUs, you can use the `nh-run-scheduler`. This tool automatically distributes runs across GPUs and starts a new one, whenever one run finishes.Calling `nh-run-scheduler` in `train` mode will train one model for each `.yml` file in a directory (or its sub-directories).```bashnh-run-scheduler train --directory /path/to/config-dir --runs-per-gpu 2 --gpu_ids 0 1 2 3 ```Use `-runs-per-gpu` to define the number of models that are simultaneously trained on a _single_ GPU (2 in this case) and `--gpu-ids` to define which GPUs will be used (numbers are ids according to nvidia-smi). In this example, 8 models will train simultaneously on 4 different GPUs.Calling `nh-run-scheduler` in `evaluate` mode will evaluate all models in all run directories in a given root directory.```bashnh-run-scheduler evaluate --directory /path/to/parent-run-dir/ --runs-per-gpu 2 --gpu_ids 0 1 2 3 ``` API usageBesides the command line tools, you can also use the NeuralHydrology package just like any other Python package by importing its modules, classes, or functions.This can be helpful for exploratory studies with trained models, but also if you want to use some of the functions or classes within a different codebase. Look at the [API Documentation](https://neuralhydrology.readthedocs.io/en/latest/api/neuralhydrology.html) for a full list of functions/classes you could use.The following example shows how to train and evaluate a model via the API.<jupyter_code>import pickle
from pathlib import Path

import matplotlib.pyplot as plt
import torch
from neuralhydrology.evaluation import metrics
from neuralhydrology.nh_run import start_run, eval_run<jupyter_output><empty_output><jupyter_text>Train a model for a single config file**Note**- The config file assumes that the CAMELS US dataset is stored under `data/CAMELS_US` (relative to the main directory of this repository) or a symbolic link exists at this location. Make sure that this folder contains the required subdirectories `basin_mean_forcing`, `usgs_streamflow` and `camels_attributes_v2.0`. If your data is stored at a different location and you can't or don't want to create a symbolic link, you will need to change the `data_dir` argument in the `1_basin.yml` config file that is located in the same directory as this notebook.- By default, the config (`1_basin.yml`) assumes that you have a CUDA-capable NVIDIA GPU (see config argument `device`). In case you don't have any or you have one but want to train on the CPU, you can either change the config argument to `device: cpu` or pass `gpu=-1` to the `start_run()` function.<jupyter_code># by default we assume that you have at least one CUDA-capable NVIDIA GPU
if torch.cuda.is_available():
    start_run(config_file=Path("1_basin.yml"))

# fall back to CPU-only mode
else:
    start_run(config_file=Path("1_basin.yml"), gpu=-1)<jupyter_output>2022-01-05 21:49:45,640: Logging to /home/frederik/Projects/neuralhydrology/examples/01-Introduction/runs/test_run_0501_214945/output.log initialized.
2022-01-05 21:49:45,641: ### Folder structure created at /home/frederik/Projects/neuralhydrology/examples/01-Introduction/runs/test_run_0501_214945
2022-01-05 21:49:45,641: ### Run configurations for test_run
2022-01-05 21:49:45,641: experiment_name: test_run
2022-01-05 21:49:45,641: train_basin_file: 1_basin.txt
2022-01-05 21:49:45,642: validation_basin_file: 1_basin.txt
2022-01-05 21:49:45,642: test_basin_file: 1_basin.txt
2022-01-05 21:49:45,642: train_start_date: 1999-10-01 00:00:00
2022-01-05 21:49:45,643: train_end_date: 2008-09-30 00:00:00
2022-01-05 21:49:45,643: validation_start_date: 1980-10-01 00:00:00
2022-01-05 21:49:45,643: validation_end_date: 1989-09-30 00:00:00
2022-01-05 21:49:45,644: test_start_date: 1989-10-01 00:00:00
2022-01-05 21:49:45,644: test_end_date: 1999-09-30 00:00:00
2022-01-05 21:49:45,644: device: cuda:0
[...]<jupyter_text>Evaluate run on test setThe run directory that needs to be specified for evaluation is printed in the output log above. Since the folder name is created dynamically (including the date and time of the start of the run) you will need to change the `run_dir` argument according to your local directory name. By default, it will use the same device as during the training process.<jupyter_code>run_dir = Path("runs/test_run_0501_214945")
eval_run(run_dir=run_dir, period="test")<jupyter_output>2022-01-05 21:53:26,501: Using the model weights from runs/test_run_0501_214945/model_epoch050.pt
# Evaluation: 100%|██████████| 1/1 [00:00<00:00,  5.20it/s]
2022-01-05 21:53:26,697: Stored results at runs/test_run_0501_214945/test/model_epoch050/test_results.p<jupyter_text>Load and inspect model predictionsNext, we load the results file and compare the model predictions with observations. The results file is always a pickled dictionary with one key per basin (even for a single basin). The next-lower dictionary level is the temporal resolution of the predictions. In this case, we trained a model only on daily data ('1D'). Within the temporal resolution, the next-lower dictionary level are `xr`(an xarray Dataset that contains observations and predictions), as well as one key for each metric that was specified in the config file.<jupyter_code>with open(run_dir / "test" / "model_epoch050" / "test_results.p", "rb") as fp:
    results = pickle.load(fp)
    
results.keys()<jupyter_output><empty_output><jupyter_text>The data variables in the xarray Dataset are named according to the name of the target variables, with suffix `_obs` for the observations and suffix `_sim` for the simulations.<jupyter_code>results['01022500']['1D']['xr']<jupyter_output><empty_output><jupyter_text>Let's plot the model predictions vs. the observations<jupyter_code># extract observations and simulations
qobs = results['01022500']['1D']['xr']['QObs(mm/d)_obs']
qsim = results['01022500']['1D']['xr']['QObs(mm/d)_sim']

fig, ax = plt.subplots(figsize=(16,10))
ax.plot(qobs['date'], qobs)
ax.plot(qsim['date'], qsim)
ax.set_ylabel("Discharge (mm/d)")
ax.set_title(f"Test period - NSE {results['01022500']['1D']['NSE']:.3f}")<jupyter_output><empty_output><jupyter_text>Next, we are going to compute all metrics that are implemented in the NeuralHydrology package. You will find additional hydrological signatures implemented in `neuralhydrology.evaluation.signatures`.<jupyter_code>values = metrics.calculate_all_metrics(qobs.isel(time_step=-1), qsim.isel(time_step=-1))
for key, val in values.items():
    print(f"{key}: {val:.3f}")<jupyter_output>NSE: 0.777
MSE: 1.099
RMSE: 1.048
KGE: 0.850
Alpha-NSE: 0.921
Beta-NSE: 0.048
Pearson-r: 0.883
FHV: -9.371
FMS: -2.198
FLV: -877.436
Peak-Timing: 0.174
{'fraction_non_alphanumeric': 0.08872712964784576, 'fraction_numerical': 0.05459374385205587, 'mean_word_length': 5.362327909887359, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 26, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '75568', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3419, 'n_tokens_neox': 2968, 'n_words': 1243, 'path': 'examples/01-Introduction/Introduction.ipynb', 'repo_name': 'DavidChoi76/neuralhydrology', 'size': '10166'}