<jupyter_start><jupyter_text>Using 3Di and Lizard together in one Jupyter notebook Welcome! In this notebook we will show how our two IT products 3Di and Lizard can be used together for an operational 3Di model. The following steps will be taken according to an example of an 3Di model:- **step 1:** Starting a 3Di simulation by using the threedi-api- **step 2:** Using radar rainfall date from Lizard for the simulation- **step 3:** Downloading the results of the 3Di-simulation- **step 4:** Analysing the results and validate with measurements from Lizard For this tutorial we make use of the "BWN-Schermer" 3Di-model:  **Step 1: Starting a 3Di simulation by using the threedi-api** Importing all required packages:<jupyter_code>from datetime import datetime
from getpass import getpass
import pandas as pd
import json
from openapi_client import ApiException, SimulationsApi, OrganisationsApi
from openapi_client.api import AuthApi
from openapi_client.api import ThreedimodelsApi
from openapi_client.models import RasterEdit
from threedi_api_client.threedi_api_client import ThreediApiClient
from pandas.io.json import json_normalize
import matplotlib.pyplot as plt
from datetime import datetime, timedelta
import requests
from pathlib import Path
import numpy as np<jupyter_output><empty_output><jupyter_text>Provide credentials to connect to the threedi-api:<jupyter_code>api_host = "https://api.3di.live/v3.0"
username = input("Username: ")
password = getpass("Password: ")
config = {
    "API_HOST": api_host,
    "API_USERNAME": username,
    "API_PASSWORD": password
}

api_client = ThreediApiClient(config=config)<jupyter_output>Username: valerie.demetriades
Password: ········<jupyter_text>Check the connection with the provided credentials:<jupyter_code>auth_api = AuthApi(api_client)

try:
    user = auth_api.auth_profile_list()
except ApiException as e:
    print("Oops, something went wrong. Maybe you made a typo?")
else:
    print(f"Successfully logged in as {user.username}!")<jupyter_output>Successfully logged in as valerie.demetriades!<jupyter_text>Let's find the model:<jupyter_code>models_api = ThreedimodelsApi(api_client)

models = models_api.threedimodels_list(name__icontains='bwn_schermer_1d2d_glg')
my_model =models.results[0]
my_model<jupyter_output><empty_output><jupyter_text>Let's see which organisations are available within my user account:<jupyter_code>organisation_api = OrganisationsApi(api_client)

organisations = organisation_api.organisations_list()


for organisation in organisations.results:
    print(f"{organisation.name}: {organisation.unique_id}")

# In this example we use the organisation from N&S Demo:
organisation_uuid = "a1993f6e13564e9687ae03a3604463f9"<jupyter_output>BWN HHNK: 48dac75bef8a42ebbb52e8f89bbdb9f2
City of Parramatta: 33b32fe8031743909ef9259744c32cc1
Course_A: 768684c9692f4a3695e014211d7fab41
Hoogheemraadschap Hollands Noorderkwartier: 474afd212f2e4b4f82615142f1d67acb
Nelen & Schuurmans: 61f5a464c35044c19bc7d4b42d7f58cb
N&S Demo: a1993f6e13564e9687ae03a3604463f9<jupyter_text>The next step is to create a simulation. Note that it will not run yet.<jupyter_code>simulation_api = SimulationsApi(api_client)

my_simulation = simulation_api.simulations_create(
    data={
        "name": "demo_3di_lizard_notebook_schermer",
        "threedimodel": my_model.id,
        "organisation": organisation_uuid,
        "start_datetime": datetime(2020, 11, 15, 14, 00),
        "duration": 3600  # in seconds, so we simulate for 1 hour
    }
)


#print an overview of the simulation
my_simulation
#check the status of the simulation with:
status = simulation_api.simulations_status_list(my_simulation.id)
print(status)<jupyter_output>{'created': datetime.datetime(2021, 3, 10, 8, 43, 49, 253381, tzinfo=tzutc()),
 'exit_code': None,
 'id': 67059,
 'name': 'created',
 'paused': None,
 'time': 0.0}<jupyter_text>We can see the simulation has not started yet. The options at the name of the status can be: "created", "started" and "finished". **Step 2: Using radar rainfall data from Lizard for the simulation** In previous step we created a simulation for the 3di model of v2_BWN Schermer. In this step we will add a rain event to the simulation and start the simulation. The rain event is extracted from the radar rainfall data within Lizard. We choose to simulate this specific date and time (15-11-2020 14:00 PM) and then extract the rainfall for 1 hour (until 15:00 PM). The simulation is also defined for 1 hour, which means the simulation stops immediately after the added rainfall event.    For this part, we also need to make a connection with the lizard api and we can use the same credentials as for the 3di part:<jupyter_code>username = input("Enter username: ")
password = getpass("Enter password: ")
headers = {
            "username": username,
            "password": password,
            "Content-Type": "application/json",
        }<jupyter_output>Enter username: valerie.demetriades
Enter password: ········<jupyter_text>Now we have to define the starttime, endtime and duration of the chosen rainfall event and the uuid of the rainfall raster within Lizard:<jupyter_code>#define start- and endtime of the radar rainfall data
start = datetime(2020, 11, 15, 14, 00)
end = start + timedelta(hours=1) #1 hour rainfall simulation
duration = end - start
print ("Starttime of the radar rainfall event:", start)
print ("Endtime of the radar rainfall event:", end)
print ("Duration of the radar rainfall event in seconds:", duration.total_seconds(), "seconds")

uuid = "730d6675-35dd-4a35-aa9b-bfb8155f9ca7"
print ("Raster rainfall uuid of Lizard:", uuid)<jupyter_output>Starttime of the radar rainfall event: 2020-11-15 14:00:00
Endtime of the radar rainfall event: 2020-11-15 15:00:00
Duration of the radar rainfall event in seconds: 3600.0 seconds
Raster rainfall uuid of Lizard: 730d6675-35dd-4a35-aa9b-bfb8155f9ca7<jupyter_text>Next step is to add this event to the created simulation in step 1 by making use of the threedi open api client:<jupyter_code>#Upload radar rainfall from Lizard and wait for it to be processed
simulation_api.simulations_events_rain_rasters_lizard_create(
            my_simulation.id,
            data={
                "offset": 0,
                "duration": int(duration.total_seconds()),
                "reference_uuid": uuid,
                "start_datetime": start,
                "units": "m/s",
            },
        )<jupyter_output><empty_output><jupyter_text>Let's print an overview of the added events to our simulation and check if the "lizardrasterrain" is not empty:<jupyter_code>events = simulation_api.simulations_events(my_simulation.id)
print(events)<jupyter_output>{'breach': [],
 'fileboundaryconditions': None,
 'filelaterals': [],
 'filerasterrain': [],
 'filerastersourcessinks': [],
 'filetimeseriesrain': [],
 'filetimeseriessourcessinks': [],
 'initial_groundwaterlevel': None,
 'initial_groundwaterraster': None,
 'initial_onedwaterlevel': None,
 'initial_onedwaterlevelpredefined': None,
 'initial_savedstate': None,
 'initial_twodwaterlevel': None,
 'initial_twodwaterraster': None,
 'initial_winddragcoefficient': None,
 'laterals': [],
 'lizardrasterrain': [{'duration': 3600,
                       'id': 2706,
                       'interval': 300,
                       'multiplier': 1.0,
                       'offset': 0,
                       'origin_offset': 0,
                       'reference_uuid': '730d6675-35dd-4a35-aa9b-bfb8155f9ca7',
                       'simulation': 'https://api.3di.live/v3.0/simulations/14809/',
                       'start_datetime': datetime.datetime(2020, 11, 15, 14, 0, tzinfo=tzutc()),
                 [...]<jupyter_text>We can now run our simulation:<jupyter_code>simulation_api.simulations_actions_create(my_simulation.id, data={"name": "start"})<jupyter_output><empty_output><jupyter_text>And check the status of the simulation with:<jupyter_code>#check the status of the simulation with:
status = simulation_api.simulations_status_list(my_simulation.id)
print(status)<jupyter_output>{'created': datetime.datetime(2021, 3, 10, 9, 51, 5, 875898, tzinfo=tzutc()),
 'exit_code': '1240 [finished]',
 'id': 67130,
 'name': 'finished',
 'paused': False,
 'time': 3600.0}<jupyter_text>**Step 3: Downloading the results of the 3Di-simulation** When our simulation has finished, it is time to analyse the results of the 3Di-model and validate the model with measurements from Lizard. Before that, we have to download all the results of the simulation. In this notebook we have already performed a 3Di simulation (with simulation id 14641), so let's select this simulation:<jupyter_code>simulation_api = SimulationsApi(api_client)

my_simulation = simulation_api.simulations_list(
    name=my_simulation.name, user__username=username).results[0]

my_simulation<jupyter_output><empty_output><jupyter_text>When our simulation has finished, it is time to analyse the results of the 3Di-model and validate the model with measurements from Lizard. Before that, we have to download all the results of the simulation: Let's see which result files are available and download all the results of the simulation model in a local folder:<jupyter_code>result_files = simulation_api.simulations_results_files_list('14809')  #fill in the id of your simulation as above

for result in result_files.results:
    print(result)
download_folder = Path(f'Results {my_simulation.name}')
download_folder.mkdir(exist_ok=True)

for file in result_files.results:
    download_url = simulation_api.simulations_results_files_download(
        id=file.id, simulation_pk=my_simulation.id
    )

    file_path = download_folder / file.filename
    r = requests.get(download_url.get_url)
    with open(file_path, 'wb') as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

    print(f"Finished downloading {file.filename}")<jupyter_output>Finished downloading results_3di.nc
Finished downloading log_files_sim_14809.zip<jupyter_text>In addition to the results of the 3Di-simulation, we also have to download the "gridadmin.h5" file of the simulation. This is a model specific file so it's under the threedimodels-api. We'll also download this file:<jupyter_code>model_api = ThreedimodelsApi(api_client)

threedi_model_id = my_simulation.threedimodel_id
download_url = model_api.threedimodels_gridadmin_download(threedi_model_id)

file_path = download_folder / "gridadmin.h5"
r = requests.get(download_url.get_url)
with open(file_path, "wb") as f:
    for chunk in r.iter_content(chunk_size=8192):
        f.write(chunk)

print(f"Finished downloading gridadmin.h5")<jupyter_output>Finished downloading gridadmin.h5<jupyter_text>All the files are now on a local folder and we can go to the next step of validation. **Analysing the results and validate with measurements from Lizard**<jupyter_code>#Import libraries
from threedigrid.admin.gridresultadmin import GridH5ResultAdmin
from threedigrid.admin.gridadmin import GridH5Admin
import os
#Define paths and files
result_path = download_folder
nc = os.path.join(result_path,'results_3di.nc')
f = os.path.join(result_path,'gridadmin.h5')

#Load files into gr object
ga = GridH5Admin(f)
gr = GridH5ResultAdmin(f,nc)<jupyter_output>[!] threedicore version differ! 
Version result file has been created with: b'2.0.16'
Version gridadmin file has been created with: 1.4.20-1<jupyter_text>Let's make a plot of the pumps within this 3Di-model:<jupyter_code># Show in a plot where these nodes are in the model
plt.figure(figsize=(10,10))
plt.plot(ga.nodes.coordinates[0], ga.nodes.coordinates[1], '.', markersize=2)
plt.plot(ga.pumps.node_coordinates[0], ga.pumps.node_coordinates[1], '*', color='red', markersize=10)

plt.axis('tight')
plt.axis('equal')

plt.xlim(112000, 120000)
plt.ylim(507000, 517000)

plt.title('Overview of the location of the pumps', fontweight='bold')<jupyter_output><empty_output><jupyter_text>Let's see the names of all pumps:<jupyter_code>ga.pumps.display_name<jupyter_output><empty_output><jupyter_text>And select the pump Beatrix:<jupyter_code>x_beatrix = ga.pumps.node_coordinates[0][-4]
y_beatrix = ga.pumps.node_coordinates[1][-4]
# Show in a plot where these nodes are in the model
plt.figure(figsize=(10,10))
plt.plot(ga.nodes.coordinates[0], ga.nodes.coordinates[1], '.', markersize=2)
plt.plot(ga.pumps.node_coordinates[0], ga.pumps.node_coordinates[1], '*', color='red', markersize=10)
plt.plot(x_beatrix, y_beatrix, '*', color='orange', markersize=20)
plt.text(x_beatrix-800, y_beatrix+200, 'Beatrix', fontsize=15, fontweight='bold')

plt.axis('tight')
plt.axis('equal')

plt.xlim(112000, 120000)
plt.ylim(507000, 517000)

plt.title('Location of Beatrix pumping station', fontweight='bold')
#connection node of this pumping station
print (gr.pumps.display_name)
pump_beatrix_connectionnode=gr.pumps.node1_id[-4]
#Select timeseries of pumping station Beatrix
pump_beatrix = gr.pumps.filter(display_name__in='Beatrix').timeseries(start_time=0,end_time=3600)
pump_beatrix.node1_id
#filter results for a connection node
model_pump_control_point = gr.nodes.filter(id=pump_beatrix_connectionnode).timeseries(start_time=0,end_time=3600)
model_wl = model_pump_control_point.s1
model_ts = [datetime.strptime(ts.split('.')[0], '%Y-%m-%dT%H:%M:%S') for ts in pump_beatrix.dt_timestamps]
model_wl
model_ts
plt.figure()

# If you want to change the size and shape of only this figure
plt.figure(figsize=(20,7))

# Plotting the timeseries of your selection
plt.plot(model_ts[1:], model_wl[1:])

# Defining the legend based on the nodes of your selection
plt.legend(['Beatrix'], frameon=True, fontsize=15, title='Pumps')

# Naming your axis
plt.xlabel('Time [seconds]')
plt.ylabel('Waterlevel [mNAP]')

# Title of your Figure
plt.title('Waterlevel at the Beatrix Pump - BWN Schermer', fontsize=20)<jupyter_output><empty_output><jupyter_text>Let's perform a validation on the waterlevels of the pumping station Beatrix. The waterlevel timeseries of this pumping station are real-time available in the datawarehouse and analysis platform Lizard.<jupyter_code>#extract the waterlevel timeseries from Lizard with the API

#Find the uuid of the timeseries you want to query
uuid = "09b5968e-9811-4539-89b3-738a4bbb6c4d"
# we're going to query request the data, similar to the simulation:
params= {'start':'2020-11-15T14:00:00Z', 'end':'2020-11-15T15:15:00Z', 'fields':'value', 'page_size':'10000000'} #Note that we use a page size of 100.
url = "https://nens.lizard.net/api/v4/timeseries/{}/events/".format(uuid)    

#Retrieve the 'results' attribute of the response using a JSON interpreter
wl_lizard = pd.DataFrame(requests.get(url=url,headers=headers, params=params).json()['results'])
#perform some conversions before plotting the timeseries
wl_lizard['datetime'] = pd.to_datetime(wl_lizard['time'])
wl_lizard = wl_lizard[['datetime', 'value']]
wl_lizard.set_index('datetime', inplace=True)

wl_lizard
plt.figure(figsize=(20,7))

#lizard
plt.plot(wl_lizard.index, wl_lizard.value)

# 3Di-model result
plt.plot(model_ts[1:], model_wl[1:])


# Defining the legend based on the nodes of your selection
plt.legend(['Lizard', '3Di'], frameon=True, fontsize=15, title='Waterlevel timeseries')

# Naming your axis
plt.xlabel('Time [seconds]')
plt.ylabel('Waterlevel [mNAP]')

# Title of your Figure
plt.title('Validation of the waterlevel at the Beatrix Pump', fontsize=20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10143707923355774, 'fraction_numerical': 0.04129984464008286, 'mean_word_length': 5.464016736401674, 'pattern_counts': {'":': 17, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '254507', 'license': 'MIT', 'n_tokens_mistral': 5508, 'n_tokens_neox': 4966, 'n_words': 1432, 'path': 'Notebooks 3Di - API v3 - VD/notebook2 - using 3Di and Lizard together in one notebook/Notebook2 - 3Di and Lizard together in one Jupyter notebook.ipynb', 'repo_name': 'threedi/scripts', 'size': '15448'}