<jupyter_start><jupyter_text>Makesense demo  Table of contents- [Introduction](introduction)- [Requierements](requierements)- [Settings](settings)- [Make](make)- [Launch](launch)- [Deploy](deploy)- [Fetch](fetch)- [Parse](parse)- [Analyze](analyze)  IntroductionThe goal of this [IPython](//ipython.org) notebook is to present how different python librairies can help orchestrate and run large scale and complex simulations campaigns. We will also see how by using the [IoTlab](//iot-lab.info) we can also reuse the exact same code  we developped for [Contiki](//contiki-os.org) operating system and push it to real nodes.We organize our code around the idea of steps that are piece of code executed independently from each others. Because of this modularity, we can put each of those steps on an IPython cell.You can launch code from this notebook that is provided inside the git repository. You can use the [fabric](//fabfile.org) command line tool from this same repository.  Requierements Python librairiesThis demo requieres some python libraries to work. All of them use the python language but you don't know to know any python to make this demo work :-)- [Networkx](//networkx.github.io) (Manipulation of graphs)- [Jinja2](//jinja.pocoo.org) (Templating)- [IPython](//ipython.org) (Interactive computation)- [pandas](//pandas.pydata.org) (Data Analysis)- [scipy](//scipy.org) (Interpolation/linear regression)- [Fabric](//fabfile.org) (launching commands)- [Numpy](//numpy.org) (numeric computation)- [requests](//python-requests.org) (HTTP requests)- [matplotlib](//matplotlib.org) (Ploting library) APT commandIf you are on a debian like system you might just get all your dependencies with this simple command:    sudo apt-get -qq install python-matplotlib fabric python-pandas python-networkx python-numpy python-jinja2 python-scipy Contiki related toolsI suppose that you got some tools that are rather very classic for Contiki development:- Contiki code source (git clone https://github.com/contiki-os/contiki.git)- mspgcc-4.7 (I use the bleeding edge version but the stable version should work)- tshark (Useful for reading PCAP files, like wireshark but with the command line)- java 1.7 (COOJA is a java app) APT command    sudo apt-get install tshark openjdk-7-jre tshark gcc-msp430     Travis-ciThe code of makesense is tested to avoid bugs. We provision a test machine (Ubuntu Machine) and then run a typical simulation inside it. To make your ubuntu machine just like the provisionned one go check the [travis configuration](//github.com/sieben/makesense/blob/master/.travis.yml)  SettingsThese settings will affect all our processes. **They are capitilized and must be changed to fit your configuration**  Global Settings<jupyter_code>import os

from os.path import join as pj
from jinja2 import Environment, FileSystemLoader

ROOT_DIR = os.getcwd()
CONTIKI_FOLDER = os.path.abspath(pj(ROOT_DIR, "contiki"))
EXPERIMENT_FOLDER = pj(ROOT_DIR, "experiments")
TEMPLATE_FOLDER = pj(ROOT_DIR, "templates")
TEMPLATE_ENV = Environment(loader=FileSystemLoader(TEMPLATE_FOLDER))
COOJA_DIR = pj(CONTIKI_FOLDER, "tools", "cooja")<jupyter_output><empty_output><jupyter_text>Simulation settingsWe are going to specify here all the settings that will affect our simulation on our host locally.<jupyter_code>name = "dummy"
platform = "wismote"
experiment_path = pj(EXPERIMENT_FOLDER, name)

# We make sure that the experiment path exists
if not os.path.exists(experiment_path):
    os.makedirs(experiment_path)

transmitting_range = 42
interference_range = 42
success_ratio_tx = 1.0
success_ratio_rx = 1.0
random_seed = 12345
timeout = 100000<jupyter_output><empty_output><jupyter_text>Make  Importing firmware from Contiki Examples<jupyter_code>from shutil import copyfile


# We can simply copy
file_to_copy = ["udp-client.c", "udp-server.c"]

for f in file_to_copy:
    copyfile(pj(CONTIKI_FOLDER, "examples", "ipv6", "rpl-udp", f),
             pj(experiment_path, f))
    
# Or we can use templating even for the firmware
makefile_template = TEMPLATE_ENV.get_template("dummy_makefile")
with open(pj(experiment_path, "Makefile"), "w") as f:
    f.write(makefile_template.render(contiki=CONTIKI_FOLDER,
                                     target="wismote"))<jupyter_output><empty_output><jupyter_text>Compiling simulator and firmware<jupyter_code>from fabric.api import lcd, local

# Making the COOJA simulator ready
with lcd(COOJA_DIR):
    print(COOJA_DIR)
    local("ant jar")

# Compiling the firmware
with lcd(experiment_path):
    local("make -j TARGET=%s clean" % platform)
    local("make -j TARGET=%s" % platform)<jupyter_output>/home/sieben/Dropbox/workspace/makesense/contiki/tools/cooja
[localhost] local: ant jar
[localhost] local: make -j TARGET=wismote clean
[localhost] local: make -j TARGET=wismote<jupyter_text>Simulation file descriptionIt's here that we are going to define the nodes topology, radio ranges, node types...You can use both the scripting abilities of python and combine them to create simulation filesvery easily.<jupyter_code># Template loading

main_csc_template = TEMPLATE_ENV.get_template("dummy_main.csc")
script_template = TEMPLATE_ENV.get_template("dummy_script.js")

# Simulation script rendering

script = script_template.render(
    timeout=timeout
)

# We can define dynamically the amount of nodes we want and their location on
# simulation. You can in a very short amount of lines test your experiment
# with a wide amount of topologies.

num_client = 7
servers = [{"mote_id": 1, "x": 0, "y": 0, "z": 0, "mote_type": "server"}]
clients = [{"mote_id": i, "x": i, "y": i, "z": 0, "mote_type": "client"} 
           for i in range(1, num_client - 1)]

# The simulation file template can also be changed and several could be written. For more information
# about how to create a campaign go check : http://makesense.readthedocs.org/en/latest/campaign.html

with open(pj(experiment_path, "main.csc"), "w") as f:
    f.write(main_csc_template.render(
        title="Dummy Simulation",
        random_seed=random_seed,
        transmitting_range=transmitting_range,
        interference_range=interference_range,
        success_ratio_tx=success_ratio_tx,
        success_ratio_rx=success_ratio_rx,
        mote_types=[
            {"name": "server", "description": "server", "firmware": "udp-server.wismote"},
            {"name": "client", "description": "client", "firmware": "udp-client.wismote"}
        ],
        motes=servers + clients,
        script=script))<jupyter_output><empty_output><jupyter_text>Launch In this step, you can launch the simulation. Our make run basically does the following thing:    java -mx512m -jar $(CONTIKI)/tools/cooja/dist/cooja.jar -nogui=main.csc -contiki=$(CONTIKI)<jupyter_code># Running the simulation locally

with lcd(experiment_path):
    local("make run")<jupyter_output>[localhost] local: make run<jupyter_text>DeployThis step is the one where we will deploy to the [iotlab](//iot-lab.info) testbed. You need to have an account and be logged. If that's not the case you checkout the following tutorial: https://www.iot-lab.info/tutorials/install-cli-tools/  Testbed settings Little SSH tipAdd the following lines to your .ssh/config while changing youruser to your propre name of course ;-)    Host strasbourg    HostName strasbourg.iot-lab.info    User leone    Host euratech    HostName euratech.iot-lab.info    User leone    Host rennes    HostName rennes.iot-lab.info    User leone    Host grenoble    HostName grenoble.iot-lab.info    User leone    Host rocquencourt    HostName rocquencourt.iot-lab.info    User leone<jupyter_code>from fabric.api import env

deploy = False

# We tell to fabric to use the .ssh/config to ease the deployment
env.use_ssh_config = True

# The testbed we want to deploy in. Because it's a simple python variable you can use loops to deploy on several testbeds.
env.host_string = "grenoble"

# Creation of folder just to be sure
testbed_results_folder = pj(experiment_path, "results", "iotlab")
if not os.path.exists(testbed_results_folder):
    os.makedirs(testbed_results_folder)<jupyter_output><empty_output><jupyter_text>Deployment settingsWe are going here to choose the nodes we want to deploy on. Looking at the GANTT chart of your testbed and position might be a good idea to have quick deployment and run. We just want to show a quick run here. You could have a much longer simulation.<jupyter_code>import json

if deploy:
    from iotlabcli import experiment

    testbed_results = pj(experiment_path, "results", "iotlab")

    # Name to give to our run
    testbed_name = "ewsn2015_demo_experiment"

    # Simulation duration in minutes
    testbed_duration = 1

    # Experiment id
    testbed_experiment_id = None

    testbed_client_nodes = [141, 142]
    testbed_server_nodes = [143]

    testbed_all_nodes = testbed_client_nodes + testbed_server_nodes

    # describe the resources
    resources = [
      {
        "nodes": [
          "m3-%d.%s.iot-lab.info" % (node, env.host_string) for node in testbed_client_nodes
        ],
        "firmware_path": pj(experiment_path, "udp-client.iotlab-m3"),
        "profile_name": "m3_energy_monitoring"
      },
      {
        "nodes": [
          "m3-%d.%s.iot-lab.info" % (node, env.host_string) for node in testbed_server_nodes
        ],
        "firmware_path": pj(experiment_path, "udp-server.iotlab-m3"),
        "profile_name": "m3_energy_monitoring"
      }
    ]

    resources = [experiment.exp_resources(**c) for c in resources]

    # Let's save all this configuration in the results folder to keep track of what we ran
    with open(pj(testbed_results_folder, "nodes.json"), "w") as f:
        f.write(json.dumps(resources,
                           sort_keys=True, indent=4, separators=(',', ': ')) )
from fabric.api import run

if deploy:
    experiment_finished = False
    
    import iotlabcli
    # gets user password from credentials file None, None
    # actually submit the experiment
    # We move to the folder to have a portable iotlab.json
    with lcd(experiment_path):
        exp_res = experiment.submit_experiment(
            api, testbed_name, testbed_duration, resources)
        testbed_experiment_id = exp_res["id"]
        with open(pj(testbed_results_folder, "experiment.json"), "w") as f:
            f.write(json.dumps({"id": testbed_experiment_id}))

    # get the content
    print("Exp submited with id: %u" % testbed_experiment_id)
    
    # We wait till the experiment is completed
    # note that this step requieres you to have experiment-cli installed on the command line.
    # TODO: Replace with a full pythonic way
    run("experiment-cli wait -i %d" % testbed_experiment_id)

    with open(pj(testbed_results_folder, "serial.log"), 'w') as f:
        run("./iot-lab/tools_and_scripts/serial_aggregator.py -i %d" % testbed_experiment_id,
            stdout=f, timeout=60 * testbed_duration + 7)
    print("Written serial logs to %s" % pj(testbed_results_folder, "serial.log"))<jupyter_output><empty_output><jupyter_text>FetchGetting all the results located on the testbed.<jupyter_code>if deploy:

    get(".iot-lab/%d" % testbed_experiment_id, local_path=testbed_results)<jupyter_output><empty_output><jupyter_text>Parse Results and logs rarely come ready to be exploited. Python is featured with regexp module and can help you createobjects representing the data you want and put them in CSV to have easier exploitation.Simulation in Cooja can have several output.- Serial log output- PCAP output- Powertracker outputAll of them can be analyzed by makesense to produce messages<jupyter_code>from makesense import parser<jupyter_output><empty_output><jupyter_text>Parse simulation results<jupyter_code># Transform all logs to a messages object

import pandas as pd
import networkx as nx
import xml.etree.ElementTree as ET
from scipy.spatial import distance
import itertools

def csc_to_graph(name):
    tree = ET.parse(name)
    l_mote_id = [int(t.text) for t in tree.findall(".//mote/interface_config/id")]
    l_mote_type = [t.text for t in tree.findall(".//mote/motetype_identifier")]
    l_x = [float(t.text) for t in tree.findall(".//mote/interface_config/x")]
    l_y = [float(t.text) for t in tree.findall(".//mote/interface_config/y")]
    l_z = [float(t.text) for t in tree.findall(".//mote/interface_config/z")]

    g = nx.Graph()
    
    # We first add all nodes with their attributes
    for (mote_id, mote_type, x, y, z) in zip(l_mote_id, l_mote_type, l_x, l_y, l_z):
        g.add_node(mote_id, mote_type=mote_type, x=x, y=y, z=z)

    # We then draw a line between all those nodes if they are within a certain range from
    # each others.
    for (a, b) in itertools.product(g.nodes(data=True), g.nodes(data=True)):
        d_a, d_b = a[1], b[1]
        if 0 < distance.euclidean((d_a["x"], d_a["y"], d_a["z"]),
                                  (d_b["x"], d_b["y"], d_b["z"])) <= transmitting_range:
            g.add_edge(a[0], b[0])

    return g
    #for mote_type in mote_types:
    #    color = mote_types[mote_type]["color"]
    #    nodelist = mote_types[mote_type]["nodes"]
    #    nx.draw_networkx_nodes(net, position,
    #                           nodelist=nodelist,
    #                           node_color=color,
    #                           ax=ax_transmission_graph)
    #nx.draw_networkx_edges(net, pos=position, ax=ax_transmission_graph)

    # labels
    #nx.draw_networkx_labels(net, position, ax=ax_transmission_graph)


messages = parser.message(experiment_path)
g = csc_to_graph(pj(experiment_path, "main.csc"))
import matplotlib
# Force matplotlib to not use any Xwindows backend.
matplotlib.use('Agg')
nx.draw(g, pos={node: [data["x"], data["y"]] for node, data in g.nodes(data=True)})<jupyter_output><empty_output><jupyter_text>Parse testbed results<jupyter_code># We were interested in the energy consumption

import csv
import numpy as np
import pandas as pd

if deploy:
    testbed_df = {}

    consumption_fields = ['junk', 'junk', 'index', 'timestamp_s', 'timestamp_us', 'power', 'voltage', 'current']

    for node in testbed_all_nodes:
        oml_name = pj(testbed_results_folder, str(testbed_experiment_id), "consumption", "m3-%d.oml" % node)
        csv_name = pj(testbed_results_folder, str(testbed_experiment_id), "consumption", "m3-%d.csv" % node)

        df = pd.read_csv(oml_name, skip_blank_lines=True, skiprows=8, names=consumption_fields, delimiter="\t")
        df.drop("junk", axis=1, inplace=True)
        df["timestamp"] = df.timestamp_s + (10 ** -6) * df.timestamp_us
        df["timestamp_diff"] = df.timestamp - df.timestamp[0]
        df.set_index("index", inplace=True)    

        # Saving to a dictionnary
        testbed_df[node] = df
        # Save the dict to a CSV for easier manipulation
        df.to_csv(csv_name)<jupyter_output><empty_output><jupyter_text>Analyze  Simulation results<jupyter_code>df_msg = pd.DataFrame([dict(msg._asdict()) for msg in messages])

# Dropna is there to clean up the columns having only None data
df_msg[df_msg.message_type == "dis"].dropna(axis=1)<jupyter_output><empty_output><jupyter_text>IoTlab results  Consumption measurementIoTlab offers access to the consumption of nodes during the experiment. Because we putted all our results inside a pandas Dataframe, and indexed it by time, we can access to all the measurement very easily.  Voltage<jupyter_code>if deploy:
    
    %matplotlib inline
    
    import matplotlib.pyplot as plt
    
    for host in testbed_all_nodes:
        testbed_df[host].voltage.plot()<jupyter_output><empty_output><jupyter_text>Power<jupyter_code>if deploy:
    
    %matplotlib inline
    
    for host in testbed_all_nodes:
        testbed_df[host].power.plot()<jupyter_output>Axes(0.125,0.125;0.775x0.775)
Axes(0.125,0.125;0.775x0.775)
Axes(0.125,0.125;0.775x0.775)<jupyter_text>Current<jupyter_code>if deploy:
    %matplotlib inline
    for host in testbed_all_nodes:
        testbed_df[host].current.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1012279498846849, 'fraction_numerical': 0.007978557626379106, 'mean_word_length': 4.08526148969889, 'pattern_counts': {'":': 23, '<': 64, '<?xml version=': 0, '>': 62, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '234250', 'license': 'Apache-2.0', 'n_tokens_mistral': 5328, 'n_tokens_neox': 5127, 'n_words': 1519, 'path': 'demo.ipynb', 'repo_name': 'sieben/makesense', 'size': '16043'}