<jupyter_start><jupyter_text>Copyright (c) 2015-2020, Swiss Federal Institute of Technology (ETH Zurich) All rights reserved.  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:  * Redistributions of source code must retain the above copyright notice, this   list of conditions and the following disclaimer.  * Redistributions in binary form must reproduce the above copyright notice,   this list of conditions and the following disclaimer in the documentation   and/or other materials provided with the distribution.  * Neither the name of the copyright holder nor the names of its   contributors may be used to endorse or promote products derived from   this software without specific prior written permission.  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.<jupyter_code>%pylab inline<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Set the notebook parameters, the path to the key file and change working-directory to the root directory of the dataprocessing framework.<jupyter_code>import os
base_dir = '/local/scratch/toolkit/'
os.environ['EXOT_ACCESS_DIR'] = str(base_dir + "benchmark_platforms_access/")
os.chdir(base_dir + 'datapro')
print("Working directory: " + str(os.getcwd()))
print("Key directory set to: " + str(os.environ['EXOT_ACCESS_DIR']))<jupyter_output>Working directory: /local/scratch/toolkit/datapro
Key directory set to: /local/scratch/toolkit/benchmark_platforms_access/<jupyter_text>Import all necessary packages.<jupyter_code>np.set_printoptions(linewidth=120)
plt.rcParams["figure.figsize"] = (18,10)
plt.rcParams["legend.loc"] = "lower right"

import toml    as TOML              # TOML library to handle config files
import pandas  as pd                # Pandas for data manipulation
import seaborn as sb                # Statistical data visualisation
import pathlib as PATH              # Used to convert strings to path objects for (de-)serialisation
import types   as tp                # Library used for online method development

# scikit-learn packets
import sklearn.base                 #
import sklearn.svm                  # LinearSVC, SVC
import sklearn.pipeline             # make_pipeline
import sklearn.preprocessing        # StandardScaler
import sklearn.decomposition        # PCA
import sklearn.naive_bayes          # GaussianNB
import sklearn.metrics              # CM

# Toolkit packets
import exot                       # Dataprocessing
import exot.experiment            # Experiment for execution
import exot.util                  # General utilities
import exot.layer                 # Data manipulation layers
from exot.util       import *
from exot.layer      import *
from exot.channel    import *
from exot.experiment import *

# Plotting
from exot.plotting.experiment_plotter import *
from exot.plotting.run_plotter        import *<jupyter_output>Tensorflow not available - excluding packets using it!<jupyter_text>Generate Experiment1. Load the configuration and define phases and environments<jupyter_code>config = TOML.load(f"configurations/examples/demo_performance_experiment_with_interference.toml")
print(TOML.dumps(config))<jupyter_output>name = "demo_performance_with_interference"
save_path = "./data"
backup_path = "./data/_backup"
experiment_exists_action = "move"

[EXPERIMENT]
type = "PerformanceExperiment"
channel = "FlushFlushCC"

[EXPERIMENT.GENERAL]
latency = 10
fan = true
governors = "userspace"
frequencies = "max"
sampling_period = 2e-5
delay_after_spawn = 5.0
delay_after_auxiliary = 1.0
active_wait = false

[EXPERIMENT.PHASES.train]
bit_count = 150
symbol_rates = "[100, 500, 5000]"
repetitions = 2

[EXPERIMENT.PHASES.eval]
bit_count = 500
symbol_rates = "[100, 500, 5000]"
repetitions = 2

[EXPERIMENT.LAYERS.src]
name = "BitsetCoding"

[EXPERIMENT.LAYERS.lne]
name = "MultiN"

[EXPERIMENT.LAYERS.rdp]
name = "DirectActivation"

[EXPERIMENT.LAYERS.io]
name = "TimeValue"

[EXPERIMENT.GENERAL.ARMv8]
fan = "255"
sampling_period = 0.000225

[EXPERIMENT.LAYERS.src.params]
bitset_length = 64

[EXPERIMENT.LAYERS.lne.params]
N = 64

[EXPERIMENT.LAYERS.rdp.params]

[EXPERIMENT.LAYERS.io.params]
timebase = "ns"

[ENVIRONMEN[...]<jupyter_text>2. Instantiate, generate and write the experiment<jupyter_code>channel = ChannelFactory()(config["EXPERIMENT"]["channel"])
experiment = ExperimentFactory()(config["EXPERIMENT"]["type"], config=config, channel=channel)

experiment.generate()
experiment.print_duration()

experiment.config.experiment_exists_action = "update"
experiment.write()<jupyter_output>[32m2020-05-22 12:58:13[0m :: bootstrapped layer 'src' with BitsetCoding
[32m2020-05-22 12:58:13[0m :: bootstrapped layer 'lne' with MultiN
[32m2020-05-22 12:58:13[0m :: bootstrapped layer 'rdp' with DirectActivation
[32m2020-05-22 12:58:13[0m :: bootstrapped layer 'io' with TimeValue
[32m2020-05-22 12:58:13[0m :: symbol rates in phase 'train' given as a str, will be evaluated
[32m2020-05-22 12:58:14[0m :: symbol rates in phase 'eval' given as a str, will be evaluated
[32m2020-05-22 12:58:15[0m :: Estimated duration of a single repetition of the train phase is 0:00:28.036600
[32m2020-05-22 12:58:15[0m :: Estimated duration of a single repetition of the eval phase is 0:00:28.097600
[32m2020-05-22 12:58:15[0m :: This results in a total estimated duration of 0:01:52.268400 for all repetitions.<jupyter_text>ExecuteRunning `execute` will execute in all environments present in the configuration. Show phases' durations.<jupyter_code>experiment.print_duration()<jupyter_output>[32m2020-05-22 12:58:23[0m :: Estimated duration of a single repetition of the train phase is 0:00:28.036600
[32m2020-05-22 12:58:23[0m :: Estimated duration of a single repetition of the eval phase is 0:00:28.097600
[32m2020-05-22 12:58:23[0m :: This results in a total estimated duration of 0:01:52.268400 for all repetitions.<jupyter_text>Finally, execute sequentially in all environments.<jupyter_code>for env in ['ARMv8', 'Haswell']:
    experiment.execute_in_environment(env, ['train', 'eval'], resume=False)
experiment.write()<jupyter_output>[32m2020-05-22 13:05:29[0m :: executing phases: ['train', 'eval'] in env 'ARMv8'
[32m2020-05-22 13:05:33[0m :: ARMv8->combined: connected: SSHUnixDriver
[32m2020-05-22 13:05:33[0m :: ARMv8->combined: sending experiment
[32m2020-05-22 13:05:34[0m :: ARMv8->combined: experiment sent!
[32m2020-05-22 13:05:34[0m :: ARMv8->combined: configuring to {'latency': 10, 'fan': '255', 'governors': 'userspace', 'frequencies': 'max', 'sampling_period': 0.000225, 'delay_after_spawn': 5.0, 'delay_after_auxiliary': 1.0, 'active_wait': False, 'STANDARD': {'latency': 10, 'fan': True, 'governors': 'userspace', 'frequencies': 'max', 'sampling_period': 2e-05, 'delay_after_spawn': 5.0, 'delay_after_auxiliary': 1.0, 'active_wait': False}}
[32m2020-05-22 13:05:43[0m :: executing in env: 'ARMv8', resume: False, reps: [0, 1], run: <Run at 0x7fa3a5d20828 (digested, config={'phase': 'train', 'bit_count': 150, 'symbol_rate': 100, 'symbol_rate_id': 0, 'repetitions': 2})>
[32m2020-05-22 13:05:43[0m :: ex[...]<jupyter_text>Analyse  Deserialisation<jupyter_code>path_to_serialised_experiment = PATH.Path(f"data/demo_performance_with_interference/_experiment.pickle")
experiment = PerformanceExperiment.read(path_to_serialised_experiment, diff_and_replace=False)<jupyter_output>[32m2020-05-22 13:14:10[0m :: unpicked an experiment instance <PerformanceExperiment at 0x7fa3a056d048 with FlushFlushCC channel, configured>
[32m2020-05-22 13:14:10[0m :: [33munpickling in a dirty git repository[0m<jupyter_text>Specify the demo run and visualise<jupyter_code>DEMO_PHASE = "train"
DEMO_ID = 0
DEMO_ARM = True
ingest_args = dict(
    lne={
        "decision_device": sklearn.pipeline.make_pipeline(
            sklearn.preprocessing.StandardScaler(), sklearn.svm.LinearSVC()
        )
    },
    rdp={"sampling_period": experiment.environment_config_general("ARMv8")["sampling_period"]}
    if DEMO_ARM
    else {},
    io={
        "env": "ARMv8" if DEMO_ARM else "Haswell",
        "rep": 0,
        "matcher": [(exot.util.wrangle.Matcher("cache", "ff", ["access_time"], list(range(64))),None)]
    },
)
demo_run = experiment.phases[DEMO_PHASE][DEMO_ID]
demo_run.ingest(**ingest_args)
with PerformanceRunPlotter(demo_run) as plotter:
    plotter.plot_rawstream(dim_count=2)
    plotter.plot_rdpstream(end=5.0, dim_count=2)
    plotter.plot_slicing(dim_count=2)
    plotter.plot_eye_diagram()
    plotter.plot_symbol_space()
    #plotter.plot_error()
    plotter.plot_timing()<jupyter_output><empty_output><jupyter_text>Analyse the experiment_Define the analysis arguments and phase mappings_<jupyter_code>analysis_args = {
    "ARMv8": dict(
        lne={
            "decision_device": sklearn.pipeline.make_pipeline(
                sklearn.preprocessing.StandardScaler(), sklearn.svm.LinearSVC()
            )
        },
        rdp={"sampling_period": experiment.environment_config_general("ARMv8")["sampling_period"]},
        io={
            "matcher": [(exot.util.wrangle.Matcher(
                "cache", "ff", ["access_time"], list(range(64))
            ),None)]
        },
    ),
    "Haswell": dict(
        lne={
            "decision_device": sklearn.pipeline.make_pipeline(
                sklearn.preprocessing.StandardScaler(), sklearn.svm.LinearSVC()
            )
        },
        io={
            "matcher": [(exot.util.wrangle.Matcher(
                "cache", "ff", ["access_time"], list(range(64))
            ),None)]
        },
    ),
}<jupyter_output><empty_output><jupyter_text>_Calculate the performance metrics_<jupyter_code>del experiment.performance_metrics
for env in ['ARMv8', 'Haswell']:
    experiment.calculate_performance_metrics(
        phase_mapping=dict([tuple(["train", "eval"])]),
        envs=[env],
        reps=[],
        **analysis_args[env],
    )<jupyter_output>[32m2020-05-22 13:15:25[0m :: analysing performance for phases: train -> eval, environments ['ARMv8']
[32m2020-05-22 13:15:37[0m :: analysing performance for phases: train -> eval, environments ['Haswell']<jupyter_text>_Show the performance metrics..._<jupyter_code>experiment.performance_metrics.head()<jupyter_output><empty_output><jupyter_text>_...and the aggregated performance metrics (mean of repetitions)_<jupyter_code>experiment.aggregate_performance_metrics()<jupyter_output><empty_output><jupyter_text>Serialise the experiment with the calculated performance metrics<jupyter_code>experiment.write()<jupyter_output><empty_output><jupyter_text>Get aggregated performance metrics for each environment and phase and output as CSV<jupyter_code>for env in ['ARMv8', 'Haswell']:
    eval_phase = 'eval'
    BERs = wrangle.filter_data(
        experiment.aggregate_performance_metrics(), environment=env, phase=eval_phase
    )
    print(env)
    print(pd.DataFrame({"bit_rate": BERs["bit_rate"], "bit_error": BERs["bit_error"]}))<jupyter_output>ARMv8
   bit_rate  bit_error
0    6400.0   0.194336
1   32000.0   0.314453
2  320000.0   0.403320
Haswell
   bit_rate  bit_error
3    6400.0   0.315848
4   32000.0   0.174944
5  320000.0   0.273717<jupyter_text>Plot performance metrics<jupyter_code>with PerformanceExperimentPlotter(experiment) as plotter:
    plotter.plot_performance_metrics()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11319198085966505, 'fraction_numerical': 0.057173500536259386, 'mean_word_length': 4.76414645744175, 'pattern_counts': {'":': 14, '<': 57, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '624016', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4467, 'n_tokens_neox': 3901, 'n_words': 984, 'path': 'notebooks/examples/demo_performance_experiment_with_interference.ipynb', 'repo_name': 'ETHZ-TEC/exot_eengine', 'size': '12121'}