<jupyter_start><jupyter_text>VOLTTRON IEEE 2030.5 (SEP2) Collector Notebook This notebook sets up a IEEE 2030.5 (SEP2.0) device driver and forwards device datafrom one VOLLTRON instance (this Collector) to another instance (the Aggregator).The Smart Energy Profile 2.0 ("SEP2") protocol implements IEEE 2030.5, and is capable of connecting a wide array of smart energy devices to the Smart Grid. The standard is designed to run over TCP/IP and is physical layer agnostic.Most of the notebook's setup and execution is done with shell commands, called from Python.  Setup: Prepare the Volttron Environment VOLTTRON must be installed before using this notebook. For detailed instructions oninstalling and configuring a VOLTTRON/Jupyter server environment, see [Jupyter Notebooks](http://volttron.readthedocs.io/en/devguides/supporting/utilities/JupyterNotebooks.html) in VOLTTRON ReadTheDocs.As is described in that guide, environment variables should have been defined before starting the Jupyter server:````$ export VOLTTRON_ROOT=~/repos/volttron````        (path of the VOLTTRON repository, installed prior to running bootstrap)````$ export VOLTTRON_HOME=~/.volttron````        (directory in which the VOLTTRON instance runs)The first VOLTTRON instance on a server usually runs, by convention, in ~/.volttron.If multiple VOLTTRON instances are to be run on a single host, each must have its own VOLTTRON_HOME.Also before starting the Jupyter server, a VOLTTRON virtual environment should have been activated by executing the following in $VOLTTRON_ROOT:````$ source env/bin/activate````The Python code below does some initialization to prepare for the steps that follow.<jupyter_code>import datetime
import json
import os
import pprint
import sqlite3
import subprocess
import sys
import time

# Define a "run this shell command" method, wrapping subprocess.check_output()
def _sh(shell_command, shell=True, stderr=None):
    try:
        return_value = subprocess.check_output(shell_command, shell=shell, stderr=stderr)
    except Exception as err:
        print('Shell command failed: {}', shell_command)
        print(err)
        return_value = 'Error'
    return return_value

# Same as _sh(), except that this also prints the command output, preceded by an optional label.
def _print_sh(shell_command, label=None, **kwargs):
    print('{0}: {1}\n'.format(label+':' if label else '', _sh(shell_command, **kwargs)))

# Set up local variables vhome and vroot.
# The environment variables VOLTTRON_ROOT and VOLTTRON_HOME should already be defined -- see above.
vroot = %env VOLTTRON_ROOT
vhome = %env VOLTTRON_HOME
print("VOLTTRON_ROOT={}".format(vroot))
print("VOLTTRON_HOME={}".format(vhome))

# Define a VIP_SOCKET environment variable for use while installing and running agents.
socket_name = 'ipc://' + vhome + '/run/vip.socket'
%env VIP_SOCKET=$socket_name

# Run from the VOLTTRON root directory.
os.chdir(vroot)

print("Initialization complete")<jupyter_output><empty_output><jupyter_text>Setup: Shut Down All Agents This ensures a clean agent installation process by the notebook.<jupyter_code>print('Wait for the list to be displayed, and confirm that no agents are listed as running...\n')

# Shut down all agents.
_sh('volttron-ctl shutdown')

# List agent status to verify that the status of each agent is 0 or blank.
_print_sh('volttron-ctl status', stderr=subprocess.STDOUT)<jupyter_output><empty_output><jupyter_text>Setup: Discover the Collector's Network Parameters In order for this Collector to forward data to an Aggregator, the Aggregatormust know the Collector's network parameters, storing them in its known_hosts file.Discover those parameters now.Copy the vip-address's IP and port, and the serverkey,to the Aggregator notebook under 'Setup: Add Each Collector to the known_hosts File',and execute that notebook's code to add this Collector to known_hosts.<jupyter_code># Obtain this server's IP address, volttron port number (usually 22916), and server key:
print('Obtaining network parameters and server key; please wait...\n')
_print_sh('curl ifconfig.me', label='Public IP address')
_print_sh('volttron-ctl auth serverkey', label='Serverkey')
_print_sh('cat {}/config'.format(vhome), label='Config file')<jupyter_output><empty_output><jupyter_text>Setup: Configure the Aggregator's Network Parameters This Collector forwards data to an Aggregator, so it must beconfigured with the Aggregator's IP address, port number and server key.Define those parameters here. Obtain them from the Aggregator notebook,'Setup: Discover the Aggregator's Network Parameters'.<jupyter_code>aggregator_vip_address = '172.16.58.3'
aggregator_vip_port = '22916'
aggregator_server_key = '<KEY>'

aggregator_vip = "tcp://{0}:{1}".format(aggregator_vip_address, aggregator_vip_port)

print('vip = {0}'.format(aggregator_vip))
print('aggregator_server_key = {0}'.format(aggregator_server_key))<jupyter_output><empty_output><jupyter_text>Setup: Test the TCP Connection The ForwardHistorian will send requests to the VOLTTRON Aggregator instancevia TCP commands. Test that the Aggregator instance is capable of receivingTCP requests on the designated IP address and port.If this test fails, the port may not be open on the other server (firewall issue?),the request may be for the wrong IP address and/or port ID,or the other server's VOLTTRON instance may be down or incorrectly configured.<jupyter_code># Use an 'nc' (netcat) command to test the TCP connection
shell_command = 'nc -z -vv -w5 {0} {1}'.format(aggregator_vip_address, aggregator_vip_port)
_print_sh(shell_command, label='Network connection test result', stderr=subprocess.STDOUT)<jupyter_output><empty_output><jupyter_text>Setup: Configure a ForwardHistorian Create a configuration file for this collector's ForwardHistorian.The file specifies the Aggregator's IP address, port and server key,and indicates which topics should be forwarded.<jupyter_code>config = """{{
    "destination-vip": "{0}",
    "destination-serverkey": "{1}",
    "required_target_agents": [],
    "custom_topic_list": [],
    "services_topic_list": ["devices"],
    "topic_replace_list": [
        {{
            "from": "FromString", 
            "to": "ToString"
        }}
    ]
}}""".format(aggregator_vip, aggregator_server_key)
print("config = {}".format(config))
config_path = vhome + '/my_ieee2030_5_forwarder.config'
with open(config_path, 'w') as file:
    file.write(config)
print('Forwarder configuration written to {}'.format(config_path))<jupyter_output><empty_output><jupyter_text>Configure a IEEE 2030.5 Device Driver Create and install a configuration for the IEEE 2030.5 device.<jupyter_code>def install_driver_csv(name=None, csv=None):
    _sh('volttron-ctl config store platform.driver {0} {1} --csv'.format(name, csv))

def install_driver_config(name=None, config=None):
    _sh('volttron-ctl config store platform.driver {0} {1}'.format(name, config))
    
# Create a CSV points file for the device
points = '''
'''
print('points file contents = {}'.format(points))

csv_path = vhome + '/my_ieee2030_5.csv'
with open(csv_path, 'w') as file:
    file.write(points)
print('IEEE 2030.5 points file written to {}\n'.format(csv_path))

# Create a config file for the device
config = """{
}"""
print("config = {}".format(config))

config_path = vhome + '/my_ieee2030_5.config'
with open(config_path, 'w') as file:
    file.write(config)
print('IEEE 2030.5 configuration written to {}\n'.format(config_path))

# Store the configurations in the platform driver.
print('\nWait for the platform driver config to display, then confirm that this config appears in it...')
install_driver_csv(name='my_ieee2030_5.csv', csv=csv_path)
install_driver_config(name='devices/my_ieee2030_5', config=config_path)

# List the driver configuration to confirm that the drivers were installed successfully.
_print_sh('volttron-ctl config list platform.driver')<jupyter_output><empty_output><jupyter_text>Setup: Install Agents Install each agent employed by the Collector: a platform driver, a IEEE 2030.5 Agent, a ForwardHistorian, and 2 Volttron Central agents.<jupyter_code>print('Wait for the list to be displayed, then confirm that all of these agents appear in it...')

def install_agent(dir=None, id=None, config=None, tag=None):
    script_install_command = 'python scripts/install-agent.py -s {0} -i {1} -c {2} -t {3} -f'
    _sh(script_install_command.format(dir, id, config, tag))
    print('Installed {}'.format(tag))

# Install the PlatformDriver agent which runs the Bacnet driver
install_agent(dir=vroot+'/services/core/PlatformDriverAgent/',
              id='platform.driver',
              config=vroot+'/services/core/PlatformDriverAgent/platform-driver.agent',
              tag='platform.driver')

# Install a ForwardHistorian agent that forwards metrics to another VOLTTRON instance
install_agent(dir=vroot+'/services/core/ForwardHistorian',
              id='forward_historian',
              config=vhome+'/my_ieee2030_5_forwarder.config',
              tag='forward_historian')

# Install a IEEE 2030.5 agent.
install_agent(dir=vroot + '/services/core/IEEE2030_5Agent',
              id='ieee2030_5agent',
              config=vroot + '/services/core/IEEE2030_5Agent/config',
              tag='ieee2030_5agent')

# Install a Platform Agent
install_agent(dir=vroot+'/services/core/VolttronCentralPlatform',
              id='platform.agent',
              config=vroot+'/services/core/VolttronCentralPlatform/config', 
              tag='vcp')

# Install a Volttron Central Agent
install_agent(dir=vroot+'/services/core/VolttronCentral',
              id='volttron.central',
              config=vroot+'/services/core/VolttronCentral/config', 
              tag='vc')

# List agent status to verify that the agents were installed successfully.
_print_sh('volttron-ctl status', stderr=subprocess.STDOUT)<jupyter_output><empty_output><jupyter_text>Setup: Get the Collector's forward_historian Credentials The Collector's ForwardHistorian agent needs to authenticate to the Aggregator. Authentication is facilitated by adding the agent's credentials to the Aggregator's auth.json file.Copy the PUBLICKEY from the command output below. On the Aggregator, run `volttron-ctl auth add` from the command line. When prompted for credentials, paste the key.<jupyter_code>_print_sh('volttron-ctl auth publickey --tag forward_historian')<jupyter_output><empty_output><jupyter_text>Execution: Refresh Variables and Stop Agents Before starting up the agents, refresh all variables and make sure that all agents are stopped.<jupyter_code>print('Make a fresh start - refresh variable definitions, shut down any running agents, refresh the database')

import datetime
import json
import os
import pprint
import sqlite3
import subprocess
import sys
import time

# Define a "run this shell command" method, wrapping subprocess.check_output()
def _sh(shell_command, shell=True, stderr=None):
    try:
        return_value = subprocess.check_output(shell_command, shell=shell, stderr=stderr)
    except Exception as err:
        print('Shell command failed: {}', shell_command)
        print(err)
        return_value = 'Error'
    return return_value

# Same as _sh(), except that this also prints the command output, preceded by an optional label.
def _print_sh(shell_command, label=None, **kwargs):
    print('{0}: {1}\n'.format(label+':' if label else '', _sh(shell_command, **kwargs)))

# Set up local variables vhome and vroot.
# The environment variables VOLTTRON_ROOT and VOLTTRON_HOME should already be defined -- see above.
vroot = %env VOLTTRON_ROOT
vhome = %env VOLTTRON_HOME
print("VOLTTRON_ROOT={}".format(vroot))
print("VOLTTRON_HOME={}".format(vhome))

# Define a VIP_SOCKET environment variable for use while installing and running agents.
socket_name = 'ipc://' + vhome + '/run/vip.socket'
%env VIP_SOCKET=$socket_name

# Run from the VOLTTRON root directory.
os.chdir(vroot)

# Shut down all agents.
_sh('volttron-ctl shutdown')

# List agent status to verify that the status of each agent is 0 or blank.
_print_sh('volttron-ctl status', stderr=subprocess.STDOUT)<jupyter_output><empty_output><jupyter_text>Execution: Start the agents<jupyter_code>print('Wait for the list to be displayed, then confirm that each started agent is running...')

_sh('volttron-ctl start --tag platform.driver')
_sh('volttron-ctl start --tag ieee2030_5agent')
_sh('volttron-ctl start --tag forward_historian')
_sh('volttron-ctl start --tag vcp')
_sh('volttron-ctl start --tag vc')

# List agent status to verify that the started agents have status "running".
_print_sh('volttron-ctl status', stderr=subprocess.STDOUT)<jupyter_output><empty_output><jupyter_text>Shutdown: Stop all agents<jupyter_code># Stop all agents.
_sh('volttron-ctl shutdown')

# Verify that all agents have been stopped.
_print_sh('volttron-ctl status', stderr=subprocess.STDOUT)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09887049357883336, 'fraction_numerical': 0.011604518025684667, 'mean_word_length': 5.2298795180722895, 'pattern_counts': {'":': 8, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '128582', 'license': 'Apache-2.0', 'n_tokens_mistral': 4050, 'n_tokens_neox': 3803, 'n_words': 1267, 'path': 'examples/JupyterNotebooks/IEEE2030_5Collector.ipynb', 'repo_name': 'lazlop/volttron', 'size': '12965'}