<jupyter_start><jupyter_text>Gas grid simulation setupThis notebook shows how the **IntegrCiTy Data Access Layer** (DAL) can be used to create a simulation setup from data stored in the 3DCityDB and how to store this setup to the 3DCityDB. In this specific example, a **simulation model** is **generated** from a representation of a gas network and energy-related data.  Creating an empty co-simulation graphWe will use [ZerOBNL](https://github.com/IntegrCiTy/zerobnl) as co-simulation platform to perform a technical assessment.Hence, we start with an empty co-simulation graph, which will be configured in the remainder of this notebook:<jupyter_code>from zerobnl import CoSim

sim = CoSim()<jupyter_output><empty_output><jupyter_text>Generating a simulation model from 3DCityDB dataCreating simulation models from CityGML data is in general no trivial task.To this end, [package dblayer](https://github.com/IntegrCiTy/dblayer) provides dedicated helper classes (contained in *dblayer.sim*) that implement a translation from CityGML data to simulation models for specific simulators.To get started, connect to the 3DCityDB:<jupyter_code>from dblayer import *

connect = PostgreSQLConnectionInfo(
    user = 'postgres',
    pwd = '<PASSWORD>',
    host = 'localhost',
    port = '5432',
    dbname = 'citydb'
    )<jupyter_output><empty_output><jupyter_text>Class *PandaNGasModelDBReader* can translate gas network models represented via the [Utility Network ADE](https://github.com/TatjanaKutzner/CityGML-UtilityNetwork-ADE) to [PandaNGas](https://github.com/IntegrCiTy/PandaNGas) models:<jupyter_code>from dblayer.sim.pandangas import *

pg_reader = PandaNGasModelDBReader( connect )
net = pg_reader.get_net( network_id = 3000 )<jupyter_output><empty_output><jupyter_text>After generetaing the model, we can take a quick peek at the model:<jupyter_code>print(net)<jupyter_output>This pandangas network includes the following parameter tables:
   - pipe (287 elements)
   - feeder (2 elements)
   - bus (277 elements)
   - load (123 elements)<jupyter_text>For futher processing below, let's extract a dict all the names of the network's loads:<jupyter_code>load_names = [ load[ 'name' ] for index, load in net.load.iterrows() ]<jupyter_output><empty_output><jupyter_text>Simulation setup for gas gridLet's add a simulator for the gas network to the co-simulation setup![ZerOBNL](https://github.com/IntegrCiTy/zerobnl) uses the concepts of *meta models*, *environments* and *nodes* to define co-simulation models (see the [documentation](https://integrcity.github.io/zerobnl/docu.html) for details).The following lines implement this concept for the gas network model:* The *meta model* is an abstraction of the actual model, defining its input and output attributes.* The *environment* provides the actual implementation of the abstract meta model.The definition of the environment includes:* A *wrapper* that serves as glue code between ZerOBNL and the simulation tool.  In this example, the wrapper will generate the simulation model from the 3DCityDB at initialization (see file [wrappers/wrapper_gasgrid.py](./wrappers/wrapper_gasgrid.py)).* A *Docker image* definition file that will be used to generate a runnable conatainer for executing the simulator (see file [dockerfiles/Dockerfile_gasgrid](./dockerfiles/Dockerfile_gasgrid)).<jupyter_code>import os.path

# Generate list of input names ("set attribute") from load names.
set_attrs = [ ( n + '/p_kW', 'kW' ) for n in load_names ]

# Define meta model.
sim.create_meta_model(
    meta_model = 'GasGridMeta',
    list_of_attrs_to_set = set_attrs,
    list_of_attrs_to_get = [] # No outputs, just inputs.
)

# Define environment.
sim.create_environment(
    env = 'GasGridEnv', 
    # The wrapper will generate the simulation model from the 3DCityDB at initialization:
    wrapper = os.path.join( 'wrappers', 'wrapper_gasgrid.py' ),
    # The Dockerfile defines the runtime environment of the simulator:
    dockerfile = os.path.join( 'dockerfiles', 'Dockerfile_gasgrid' )
)

sim.add_node(
    node = 'GasGrid', 
    meta = 'GasGridMeta',
    env = 'GasGridEnv',
    # These initial values will be used to connect to the 3DCityDB and load the correct network model:
    init_values = {   
        'dbuser': 'postgres',
        'dbpwd': '<PASSWORD>',
        'dbhost': 'citydb-container',
        'dbname': 'citydb',
        'dbport': 5432,
        'network_id': 3000
    }
)<jupyter_output><empty_output><jupyter_text>Simulation setup for consumersNext, add a component to the co-simulation setup that represents the customers. In this simple example, it will only read all energy consumption data from a file.The 3DCityDB contains all the data needed to link the sinks of the gas network with consumption data from building.In this specific case, it goes as follows:1. Energy consumption data (stored as time series) is linked to objects of type *FinalEnergy*.2. Object of type *FinalEnergy* are associated to objects of type *EnergyConversionSystem* (in this case, they are boilers).3. Objects of type *EnergyConversionSystem* are linked to buildings.4. Buildings are linked to objects of type *TerminalElement* (belonging to networks).5. Objects of type *TerminalElement* are linked to objects of type *NetworkFeature* of the same network (which correspond to network nodes of the simulation models).Once you know how the data is linked in the 3DCityDB, you can use [package dblayer](https://github.com/IntegrCiTy/dblayer) to retrieve this data.A lot of relevant information from separate tables is already joind in *views*, and can be retrieved directly.But beyond that, the user has to define queries that connect entries via relations of their attributes.The following lines show how a complex database query can be defined, joining conditions over several database tables and views.First, we retrieve the definition of the data types that we want to work with:<jupyter_code>db_access = DBAccess()
db_access.connect_to_citydb(connect)

# Get definition of type "FinalEnergy" from view "nrg8_final_energy_ts". 
# This view already provides the link of objects to time series data.
FinalEnergy = db_access.map_citydb_object_class( 
    'FinalEnergy', table_name='nrg8_final_energy_ts', schema='citydb_view' 
)

# Get definition of linking between objects of type "FinalEnergy" and 
# "EnergyConversionSystem" from table "nrg8_conv_sys_to_final_nrg".
ConvSystemToFinalEnergy = db_access.map_citydb_object_class( 
    'ConvSystemToFinalEnergy', 
    table_name='nrg8_conv_sys_to_final_nrg', schema='citydb', user_defined = True
)

# Get definition of type "EnergyConversionSystem" from table "nrg8_conv_system".
# Call them "Boiler" (because that is what they are in this case).
Boiler = db_access.map_citydb_object_class( 
    'Boiler', 
    table_name='nrg8_conv_system', schema='citydb' 
)

# Get definition of type "TerminalElement" from view "utn9_ntw_feat_term_elem".
# This view already provides the link of objects to associated network features.
TerminalElement = db_access.map_citydb_object_class( 
    'TerminalElement', 
    table_name='utn9_ntw_feat_term_elem', schema='citydb_view' 
)

# Get definition of linking between objects of type "NetworkFeature" to objects 
# of type "Network" from table "utn9_network_to_network_feature". 
# This linking has no official name, so we call it "NetworkToFeature".
NetworkToFeature = db_access.map_citydb_object_class( 
    'NetworkToFeature',
    table_name='utn9_network_to_network_feature', schema='citydb', user_defined = True
)<jupyter_output><empty_output><jupyter_text>Now make the actual query to retrieve the data:<jupyter_code>final_energy_db_data = db_access.join_citydb_objects(
    # Only retrieve data of related to "FinalEnergy" and "TerminalElement".
    [ 'FinalEnergy', 'TerminalElement' ],
    # Define the logical relations between all these types and their attributes.
    conditions = [
        FinalEnergy.id == ConvSystemToFinalEnergy.final_nrg_id,
        ConvSystemToFinalEnergy.conv_system_id == Boiler.id,
        Boiler.inst_in_ctyobj_id == TerminalElement.conn_cityobject_id,
        TerminalElement.id == NetworkToFeature.network_feature_id,
        NetworkToFeature.network_id == 3000        
        ]
    )<jupyter_output><empty_output><jupyter_text>Now extract the data that we actually want: the **mapping** of **network node names** to **time series**.<jupyter_code># Each results contains an instance of "FinalEnergy" (index 0) and "TerminalElement" (index 1).
# From that, the following line extracts a dict of network node names and time series.
gas_demand = { data[1].name: data[0].ts_values_array for data in final_energy_db_data }

# Also extract start date, interval length and unit as well as number of entries for all time series.
# Get only for the first, same for all others (in this example).
time_series_start_date = final_energy_db_data[0][0].ts_temporal_extent_begin
time_series_interval = final_energy_db_data[0][0].ts_time_interval
time_series_interval_unit = final_energy_db_data[0][0].ts_time_interval_unit
time_series_length = final_energy_db_data[0][0].ts_array_length<jupyter_output><empty_output><jupyter_text>Convert the data to a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.htmldataframe):<jupyter_code>import pandas as pd

df = pd.DataFrame.from_dict(gas_demand)

date = pd.date_range(
    start = time_series_start_date, 
    periods = time_series_length, 
    freq = '{length}{unit}'.format( length=time_series_interval, unit=time_series_interval_unit )
)

df['date'] = date.values
df.set_index('date', inplace=True)<jupyter_output><empty_output><jupyter_text>Store the DataFrame as [pickle file](https://docs.python.org/3/library/pickle.html) (to be used by the simulator).<jupyter_code>df.to_pickle( './gas_demand.pkl' )<jupyter_output><empty_output><jupyter_text>Let's not hardcode the start date of the time series as a parameter for the simulation model, but rather look-up the value from the database during the initialization of the simulation.This can be done by passing an intance of class *AssociateCityDBObjectAttribute* as parameter to ZerOBNL's list of initial paramters, which will be resolved at runtime:<jupyter_code>from dblayer.zerobnl.associate import *

# Since all time series in this example have the same start time, we just pass a reference to first in the list.
time_series = final_energy_db_data[0][0]

# Make association to the object's attribute "ts_temporal_extent_begin".
start_date = AssociateCityDBObjectAttribute( time_series, 'ts_temporal_extent_begin' )<jupyter_output><empty_output><jupyter_text>Like above for the gas network simulation model, the *meta model*, *environment* and *node* needs to be defined.Similar to above, this includes again the wrapper (see file [wrappers/wrapper_consumers.py](./wrappers/wrapper_consumers.py)).and the Dockerfile (see file [dockerfiles/Dockerfile_consumers](./dockerfiles/Dockerfile_consumers)).<jupyter_code># Generate list of output names ("get attribute") from load names.
get_attrs = [ ( n, 'kW' ) for n in load_names ]

# Define meta model.
sim.create_meta_model(
    meta_model = 'ConsumersMeta',
    list_of_attrs_to_set = [], # No inputs, just outputs.
    list_of_attrs_to_get = get_attrs
)

# Define environment.
sim.create_environment(
    env = 'ConsumersEnv', 
    wrapper = os.path.join( 'wrappers', 'wrapper_consumers.py' ),
    dockerfile = os.path.join( 'dockerfiles', 'Dockerfile_consumers' )
)

# Define node.
sim.add_node(
    node = 'Consumers', 
    meta = 'ConsumersMeta',
    env = 'ConsumersEnv',
    init_values = { 
        # Pass the association object from above, to look-up the value 
        # from the database during the initialization of the simulation.
        'start_date': start_date 
    },
    # Add the pickle file from above as additional resource.
    files = [ './gas_demand.pkl' ]
)<jupyter_output><empty_output><jupyter_text>LinksAdd the links between the individual co-simulation components.In this simple case, this is just the connection of all outputs of the consumers model (node *Consumers*) to all inputs of the gas network model (node *GasGrid*):<jupyter_code>for load in load_names:
    sim.add_link( 'Consumers', load, 'GasGrid', '{}/p_kW'.format( load ) )<jupyter_output><empty_output><jupyter_text>We can view the generated links to check that they are correct:<jupyter_code>sim.links<jupyter_output><empty_output><jupyter_text>Simulation parametersFinally, add the following parameters to the setup:* **Sequence**: For each step, simulate node *Consumers* first, then node *GasGrid*.* **Steps**: List of synchronization steps (time differences between two simulation steps).* **Time Unit**: Time unit of synchronization steps.<jupyter_code>sim.create_sequence( [ [ 'Consumers' ], [ 'GasGrid' ] ] )
sim.create_steps( 12 * [60*60] )
sim.set_time_unit( 'seconds' )<jupyter_output><empty_output><jupyter_text>Save the simulation setup to the databaseFor saving ZerOBNL simulation setups to the database, [package dblayer](https://github.com/IntegrCiTy/dblayer) provides a dedicated helper class (*dblayer.zerobnl*):<jupyter_code>from dblayer.zerobnl.writer import *

writer = DBWriter( connect )
writer.write_to_db( sim, 'GasGridSim', write_meta_models = True, write_envs = True )<jupyter_output><empty_output><jupyter_text>Finally, delete the instance of class DBAccess to close the session.<jupyter_code>del writer<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08743331357439241, 'fraction_numerical': 0.00555720213396562, 'mean_word_length': 5.354519774011299, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '35292', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4160, 'n_tokens_neox': 3999, 'n_words': 1337, 'path': '3_simulate/3a_sim_setup.ipynb', 'repo_name': 'IntegrCiTy/DemoToolchain', 'size': '13492'}