<jupyter_start><jupyter_text>Adding New WorkloadsAdding new workloads into benchmark-wrapper is a fairly straightforward process, but requires a bit of work. This page tracks all the changes that a user needs to make when adding in a new benchmark.    benchmark-wrapper is currently undergoing a re-write, meaning a lot of change is happening pretty quickly. The information on this page is relevant to the new modifications which change the way in which benchmarks should be added. It may not be consistent to the way in which existing benchmarks are developed.    This page is written within a Jupyter Notebook, however running benchmark-wrapper from within a Jupyter Notebook is not tested nor supported. Step Zero: PrepA Benchmark within benchmark-wrapper is essentially just a Python module which handles setting up, running, parsing and tearing down a benchmark. To create our benchmark, we need to understand the following items:1. What is the human-readable, camel-case-able string name for our benchmark?1. What arguments does the benchmark wrapper need from the user?1. Are there any setup tasks that our benchmark wrapper needs to perform?1. How do we run our benchmark?1. Are there any cleanup tasks that our benchmark wrapper needs to perform?1. What data should the benchmark export?In this example, we'll create a new benchmark wrapper that does a ping test against a list of given hostnames and IPs:1. We'll call it ``pingtest``1. We need to know which hosts the user wants to ping and how many pings the user wants to perform.1. We need to verify that the arguments that the user gave us are valid. We'll also create a temp file to show that the benchmark is running.1. We can run our ping tests using the ``ping`` shell command.1. We need to clean up our 'I-am-running' temp file.1. For each pinged host, we want to output a single result detailing the result of the ping session (RTT information, packet loss %, IP resolution, errors). Step One: Initialize To begin, let's create the required files for our benchmark by creating a new Python package under ``snafu/benchmarks``:```textsnafu/benchmarks/pingtest/├── __init__.py└── pingtest.py```Inside ``pingtest.py``, we'll create our initial ``Benchmark`` subclass. Inside this subclass, there are a fewclass variables which we need to set:1. ``tool_name``: This is the camel-case name for our benchmark.1. ``args``: These are the arguments which our Benchmark will pull from the user through the CLI, OS environment, and/or from a configuration file (CLI is preferred over the OS environment, which is preferred over the configuration file). In the background, snafu uses [configargparse](https://pypi.org/project/ConfigArgParse/) to do the dirty-work, which is a helpful wrapper around Python's own [argparse](https://docs.python.org/3/library/argparse.html). The ``args`` class variable should be set to a tuple of ``snafu.config.ConfigArgument``s, which take in arguments just like ``configargparse.ArgumentParser.add_argument``. If you have used argparse in the past, this should look super familiar.1. ``metadata``: This is an iterable of strings which represent the metadata that will be exported with the Benchmark's results. Each string corresponds to the attribute name that the argument is stored under by ``configargparse``. For instance, creating a new argument under the ``args`` class variable with ``"--my-metadata", dest="mmd"`` would result in the attribute name being ``mmd``. Adding ``mmd`` under ``metadata`` will in turn cause the value for the ``--my-metadata`` argument to be exported as metadata. Benchmarks will by default specify ``cluster_name``, ``user`` and ``uuid`` arguments as metadata, but if you want to use your own set of metadata keys it can be set here.<jupyter_code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Ping hosts and export results."""
from snafu.config import ConfigArgument
from snafu.benchmarks import Benchmark


class PingTest(Benchmark):
    """Wrapper for the Ping Test benchmark."""

    tool_name = "pingtest"
    args = (
        ConfigArgument(
            "--host",
            help="Host(s) to ping. Can give more than one host by separating them with "
                 "spaces on the CLI and in config files, or by giving them in a "
                 "pythonic-list format through the OS environment",
            dest="host",
            nargs="+",
            env_var="HOST",
            type=str,
            required=True,
        ),
        ConfigArgument(
            "--count",
            help="Number of pings to perform per sample.",
            dest="count",
            env_var="COUNT",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--samples",
            help="Number of samples to perform.",
            dest="samples",
            env_var="SAMPLES",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--htlhcdtwy",
            help="Has The Large Hadron Collider Destroyed The World Yet?",
            dest="htlhcdtwy",
            env_var="HTLHCDTWY",
            default="no",
            type=str,
            choices=["yes", "no"]
        ),
    )
    # don't care about Cluster Name, but the Hadron Collider is serious business
    metadata = ("user", "uuid", "htlhcdtwy")

    def setup(self):
        """Setup the Ping Test Benchmark."""
        pass

    def collect(self):
        """Run the Ping Test Benchmark and collect results."""
        pass

    def cleanup(self):
        """Cleanup the Ping Test Benchmark."""
        pass<jupyter_output><empty_output><jupyter_text>Let's check that we're ready to move on by trying to parse some configuration parameters. Let's load up Python!benchmark-wrapper includes a special variable called ``snafu.registry.TOOLS`` which will map a benchmark'scamel-case string name to its wrapper class. Let's use this to create an instance of our benchmark andparse some configuration.<jupyter_code>from snafu.registry import TOOLS
from pprint import pprint
pingtest = TOOLS["pingtest"]()

# Set some config parameters
# Config file
!echo "samples: 3" > my_config.yaml
!echo "count: 5" >> my_config.yaml
# OS ENV
import os
os.environ["HOST"] = "[www.google.com,www.bing.com]"

# Parse arguments and print result
# Since we aren't running within the main script (run_snafu.py),
# need to add the config option manually
pingtest.config.parser.add_argument("--config", is_config_file=True)
pingtest.config.parse_args(
    "--config my_config.yaml --labels=notebook=true --uuid 1337 --user snafu "
    "--htlhcdtwy=no".split(" ")
)
pprint(vars(pingtest.config.params))

del pingtest
!rm my_config.yaml<jupyter_output><empty_output><jupyter_text>Now that we have our configuration all ready to go, let's start filling in our benchmark.  Step Two: Setup MethodEach benchmark is expected to have a ``setup`` method, which will return ``True`` if setup tasks completed successfully and otherwise return ``False``.For our use case, let's write a file to ``/tmp`` that can signal other programs that our benchmark is running. We'll also check if our temporary file exists before writing it, which would indicate that something is wrong.    Remeber, we are still working in our module found at snafu/benchmarks/pingtest.py<jupyter_code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Ping hosts and export results."""
from snafu.config import ConfigArgument
from snafu.benchmarks import Benchmark

# We'll also import this helpful function from the config module
import os
from snafu.config import check_file


class PingTest(Benchmark):
    """Wrapper for the Ping Test benchmark."""

    tool_name = "pingtest"
    args = (
        ConfigArgument(
            "--host",
            help="Host(s) to ping. Can give more than one host by separating them with "
                 "spaces on the CLI and in config files, or by giving them in a "
                 "pythonic-list format through the OS environment",
            dest="host",
            nargs="+",
            env_var="HOST",
            type=str,
            required=True,
        ),
        ConfigArgument(
            "--count",
            help="Number of pings to perform per sample.",
            dest="count",
            env_var="COUNT",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--samples",
            help="Number of samples to perform.",
            dest="samples",
            env_var="SAMPLES",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--htlhcdtwy",
            help="Has The Large Hadron Collider Destroyed The World Yet?",
            dest="htlhcdtwy",
            env_var="HTLHCDTWY",
            default="no",
            type=str,
            choices=["yes", "no"]
        ),
    )
    # don't care about Cluster Name, but the Hadron Collider is serious business
    metadata = ("user", "uuid", "htlhcdtwy")
    
    TMP_FILE_PATH = "/tmp/snafu-pingtest"

    def setup(self) -> bool:
        """
        Setup the Ping Test Benchmark.
        
        This method creates a temporary file at ``/tmp/snafu-pingtest`` to let others 
        know that the benchmark is currently running.
        
        Returns
        -------
        bool
            True if the temporary file was created successfully, othewise False. Will
            also return False if the temporary file already exists.
        """
        
        if check_file(self.TMP_FILE_PATH):
            # The benchmark base class exposes a logger at self.logger which we can use
            self.logger.critical(
                f"Temporary file located at {self.TMP_FILE_PATH} already exists."
            )
            return False
        
        try:
            tmp_file = open(self.TMP_FILE_PATH, "x")
            tmp_file.close()
        except Exception as e:
            self.logger.critical(
                f"Unable to create temporary file at {self.TMP_FILE_PATH}: {e}",
                exc_info=True
            )
            
            return False
        else:
            self.logger.info(
                f"Successfully created temp file at {self.TMP_FILE_PATH}"
            )
            return True

    def collect(self):
        """Run the Ping Test Benchmark and collect results."""
        pass

    def cleanup(self):
        """Cleanup the Ping Test Benchmark."""
        pass<jupyter_output><empty_output><jupyter_text>Let's test it out and make sure our setup method works properly:<jupyter_code>from snafu.registry import TOOLS
pingtest = TOOLS["pingtest"]()

!rm -f /tmp/snafu-pingtest

# No file exists
print(f"Setup result is: {pingtest.setup()}")

# File exists
print(f"Setup result is: {pingtest.setup()}")

# Create failure in open
!rm -f /tmp/snafu-pingtest
open_bak = open
open = lambda file, mode: int("I'm a string")
print(f"Setup result is: {pingtest.setup()}")

# Cleanup
open = open_bak
!rm -f /tmp/snafu-pingtest
del pingtest<jupyter_output><empty_output><jupyter_text>Step Three: Cleanup MethodNow let's go ahead and populate our cleanup method. The ``cleanup`` method has the same usage as the setup method: return ``True`` if the cleanup was successfull, otherwise ``False``. For the ping test benchmark, we just need to remove our temporary file:<jupyter_code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Ping hosts and export results."""
import os
from snafu.config import ConfigArgument, check_file
from snafu.benchmarks import Benchmark


class PingTest(Benchmark):
    """Wrapper for the Ping Test benchmark."""

    tool_name = "pingtest"
    args = (
        ConfigArgument(
            "--host",
            help="Host(s) to ping. Can give more than one host by separating them with "
                 "spaces on the CLI and in config files, or by giving them in a "
                 "pythonic-list format through the OS environment",
            dest="host",
            nargs="+",
            env_var="HOST",
            type=str,
            required=True,
        ),
        ConfigArgument(
            "--count",
            help="Number of pings to perform per sample.",
            dest="count",
            env_var="COUNT",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--samples",
            help="Number of samples to perform.",
            dest="samples",
            env_var="SAMPLES",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--htlhcdtwy",
            help="Has The Large Hadron Collider Destroyed The World Yet?",
            dest="htlhcdtwy",
            env_var="HTLHCDTWY",
            default="no",
            type=str,
            choices=["yes", "no"]
        ),
    )
    # don't care about Cluster Name, but the Hadron Collider is serious business
    metadata = ("user", "uuid", "htlhcdtwy")
    
    TMP_FILE_PATH = "/tmp/snafu-pingtest"

    def setup(self) -> bool:
        """
        Setup the Ping Test Benchmark.
        
        This method creates a temporary file at ``/tmp/snafu-pingtest`` to let others 
        know that the benchmark is currently running.
        
        Returns
        -------
        bool
            True if the temporary file was created successfully, othewise False. Will
            also return False if the temporary file already exists.
        """
        
        
        if check_file(self.TMP_FILE_PATH):
            # The benchmark base class exposes a logger at self.logger which we can use
            self.logger.critical(
                f"Temporary file located at {self.TMP_FILE_PATH} already exists."
            )
            return False
        
        try:
            tmp_file = open(self.TMP_FILE_PATH, "x")
            tmp_file.close()
        except Exception as e:
            self.logger.critical(
                f"Unable to create temporary file at {self.TMP_FILE_PATH}: {e}",
                exc_info=True
            )
            
            return False
        else:
            self.logger.info(
                f"Successfully created temp file at {self.TMP_FILE_PATH}"
            )
            return True

    def collect(self):
        """Run the Ping Test Benchmark and collect results."""
        pass

    def cleanup(self) -> bool:
        """
        Cleanup the Ping Test Benchmark.
        
        This method removes the temporary file at ``/tmp/snafu-pingtest`` to let others
        know that the benchmark has finished running.
        
        Returns
        -------
        bool
            True if the temporary file was deleted successfully, otherwise False.
        """
        
        try:
            os.remove(self.TMP_FILE_PATH)
        except Exception as e:
            self.logger.critical(
                f"Unable to remove temporary file at {self.TMP_FILE_PATH}: {e}",
                exc_info=True
            )
            
            return False
        else:
            self.logger.info(
                f"Successfully removed temp file at {self.TMP_FILE_PATH}"
            )
            return True<jupyter_output><empty_output><jupyter_text>And again, some quick tests just to verify it works as expected:<jupyter_code>from snafu.registry import TOOLS
pingtest = TOOLS["pingtest"]()

!rm -f /tmp/snafu-pingtest

# No file exists, so should error
print(f"Cleanup result is {pingtest.cleanup()}")

# Create the file using setup(), then cleanup()
print(f"Setup result is {pingtest.setup()}")
print(f"Cleanup result is {pingtest.cleanup()}")

# Cleanup
del pingtest<jupyter_output><empty_output><jupyter_text>Now we have our setup and cleanup methods good to go, let's get to the fun part.  Part Four: Collect MethodThe collect method is an iterable that returns a special dataclass that is shipped with benchmark-wrapper, called a ``BenchmarkResult``. BenchmarkResult holds important information about a benchmark's resulting data, such as the configuration, metadata, labels and numerical data. It also understands how to prepare itself for export. All Benchmarks are expected to return their results using this dataclass in order to support a common interface for data exporters, reduce code reuse, and reduce extra overhead.The base Benchmark class includes a helpful method called ``create_new_result``, which we will use in the example below.For our ping test benchmark, our collect method needs to run the ping command, parse its output, and yield a new BenchmarkResult. To help prevent the collect method itself from becoming super large and out of control, we'll create some new methods in our wrapper class that the collect method will call to help do its thing. Benchmarks can have any number of additional methods, just as long as they have setup, collect and cleanup.One last note here before the code: benchmark-wrapper ships with another helpful module called ``process``, which contains functions and classes to facilitate running subprocesses. In particular, we'll be using the ``LiveProcess`` wrapper.<jupyter_code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Ping hosts and export results."""
import os
from snafu.config import ConfigArgument, check_file
from snafu.benchmarks import Benchmark

# Grab the LiveProcess class, the BenchmarkResult class, stuff
# for type hints, and dataclasses for storing our ping results
from snafu.process import LiveProcess, ProcessRun
from snafu.benchmarks import BenchmarkResult
from typing import Iterable, Optional
from dataclasses import dataclass, asdict
# Also shlex and subprocess for creating our ping command
import shlex
import subprocess
# And finally re for regex
import re


@dataclass
class PingResult:
    ip: Optional[str] = None
    success: Optional[bool] = None
    fail_msg: Optional[str] = None
    host: Optional[str] = None
    transmitted: Optional[int] = None
    received: Optional[int] = None
    packet_loss: Optional[float] = None
    packet_bytes: Optional[int] = None
    time_ms: Optional[float] = None
    rtt_min_ms: Optional[float] = None
    rtt_avg_ms: Optional[float] = None
    rtt_max_ms: Optional[float] = None
    rtt_mdev_ms: Optional[float] = None


class PingTest(Benchmark):
    """Wrapper for the Ping Test benchmark."""

    tool_name = "pingtest"
    args = (
        ConfigArgument(
            "--host",
            help="Host(s) to ping. Can give more than one host by separating them with "
                 "spaces on the CLI and in config files, or by giving them in a "
                 "pythonic-list format through the OS environment",
            dest="host",
            nargs="+",
            env_var="HOST",
            type=str,
            required=True,
        ),
        ConfigArgument(
            "--count",
            help="Number of pings to perform per sample.",
            dest="count",
            env_var="COUNT",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--samples",
            help="Number of samples to perform.",
            dest="samples",
            env_var="SAMPLES",
            default=1,
            type=int,
        ),
        ConfigArgument(
            "--htlhcdtwy",
            help="Has The Large Hadron Collider Destroyed The World Yet?",
            dest="htlhcdtwy",
            env_var="HTLHCDTWY",
            default="no",
            type=str,
            choices=["yes", "no"]
        ),
    )
    # don't care about Cluster Name, but the Hadron Collider is serious business
    metadata = ("user", "uuid", "htlhcdtwy")
    
    TMP_FILE_PATH = "/tmp/snafu-pingtest"

    HOST_RE = r"PING ([a-zA-Z0-9.-]+) \(([0-9.]+)\) \d+\(([\d]+)\) bytes of data\."
    RTT_STATS_RE = r"rtt min\/avg\/max\/mdev = ([\d.]+)\/([\d.]+)\/([\d.]+)\/([\d.]+) ms"
    PACKET_RE = r"(\d+) packets transmitted, (\d+) received, ([\d.]+)\% packet loss, " \
                r"time (\d+)ms"
    
    def setup(self) -> bool:
        """
        Setup the Ping Test Benchmark.
        
        This method creates a temporary file at ``/tmp/snafu-pingtest`` to let others 
        know that the benchmark is currently running.
        
        Returns
        -------
        bool
            True if the temporary file was created successfully, othewise False. Will
            also return False if the temporary file already exists.
        """
        
        
        if check_file(self.TMP_FILE_PATH):
            # The benchmark base class exposes a logger at self.logger which we can use
            self.logger.critical(
                f"Temporary file located at {self.TMP_FILE_PATH} already exists."
            )
            return False
        
        try:
            tmp_file = open(self.TMP_FILE_PATH, "x")
            tmp_file.close()
        except Exception as e:
            self.logger.critical(
                f"Unable to create temporary file at {self.TMP_FILE_PATH}: {e}",
                exc_info=True
            )
            
            return False
        else:
            self.logger.info(
                f"Successfully created temp file at {self.TMP_FILE_PATH}"
            )
            return True
    
    def parse_host_info(self, stdout: str, store: PingResult) -> None:
        """
        Parse the host line of ping stdout.
        
        Expected format is: ``PING host (ip) data_bytes(ICMP_data_bytes) ...``.
        
        Parameters
        ----------
        stdout : str
            ping stdout to parse
        store : PingResult
            PingResult instance to store parsed variables into
        """
        
        result = re.compile(self.HOST_RE).search(stdout)
        if result is None:
            self.logger.warning(f"Unable to parse host info!")
            return
        host, ip, data_size = result.groups()
        data_size = int(data_size)
        
        if host == ip:
            host = None  # user pinged an IP rather than a host
        
        store.host = host
        store.ip = ip
        store.packet_bytes = data_size
    
    def parse_packet_stats(self, stdout: str, store: PingResult) -> None:
        """
        Parse the packet statistics line of ping stdout.
        
        Expected format is: 
        ``A packets transmitted, B received, C% packet loss, time Dms``
        
        Parameters
        ----------
        stdout : str
            ping stdout to parse
        store : PingResult
            PingResult instance to store parsed variables into
        """
        
        result = re.compile(self.PACKET_RE).search(stdout)
        if result is None:
            self.logger.warning(
                f"Unable to parse packet stats!"
            )
            return
        transmitted, received, packet_loss, time_ms = map(int, result.groups())

        store.transmitted = transmitted
        store.received = received
        store.packet_loss = packet_loss
        store.time_ms = time_ms
    
    def parse_rtt_stats(self, stdout: str, store: PingResult) -> None:
        """
        Parse the RTT statistics line of ping stdout.
        
        Expected format is: ``rtt min/avg/max/mdev = A/B/C/D ms``
        
        Parameters
        ----------
        stdout : str
            ping stdout to parse
        store : PingResult
            PingResult instance to store parsed variables into
        """
        
        result = re.compile(self.RTT_STATS_RE).search(stdout)
        if result is None:
            self.logger.warning(f"Unable to parse rtt stats!")
            return
        rtt_min, rtt_avg, rtt_max, rtt_mdev = map(float, result.groups())
        store.rtt_min_ms = rtt_min
        store.rtt_avg_ms = rtt_avg
        store.rtt_max_ms = rtt_max
        store.rtt_mdev_ms = rtt_mdev
        
    def parse_stdout(self, stdout: str) -> PingResult:
        """
        Parse the stdout of the ping command.
        
        Tested against ping from iputils 20210202 on Fedora Linux 34
        
        Returns
        -------
        PingResult
        """
        
        # We really only care about the first line, and the last two lines
        lines = stdout.strip().split("\n")
        
        # Check if we got an error
        if len(lines) == 1:
            msg = lines[0]
            return PingResult(
                fail_msg=msg,
                success=False
            )
        
        
        result = PingResult(success=True)
        self.parse_host_info(stdout, result)
        self.parse_packet_stats(stdout, result)
        self.parse_rtt_stats(stdout, result)
        
        return result
    
    def ping_host(self, host: str) -> Iterable[BenchmarkResult]:
        """
        Run the ping test benchmark against the given host.
        
        Parameters
        ----------
        host : str
            Host to ping
        
        Returns
        -------
        iterable
            Iterable of BenchmarkResults
        """
        
        self.logger.info(f"Running ping test against host {host}")
        cmd = shlex.split(f"ping -c {self.config.count} {host}")
        self.logger.debug(f"Using command: {cmd}")
        
        # A config instance allows for accessing params directly,
        # therefore self.config.samples == self.config.params.samples
        for sample_num in range(self.config.samples):
            self.logger.info(f"Collecting sample {sample_num}")
            
            # We'll use the LiveProcess context manager to run ping
            # LiveProcess will expose the Popen object at 'process',
            # create a queue with lines from stdout at 'stdout',
            # and create a snafu.process.ProcessRun instance at `attempt`
            
            # Here we will tell LiveProcess to send stdout and stderr
            # to the same pipe
            with LiveProcess(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as lp:
                # if process hasn't finished
                while lp.process.poll() is None:
                    # if we have a new line in stdout to get
                    if not lp.stdout.empty():
                        self.logger.debug(lp.stdout.get())
                # when we get here, the process has finished
                process_run: ProcessRun = lp.attempt
            
            self.logger.debug(f"Got process run: {vars(process_run)}")
            result: PingResult = self.parse_stdout(process_run.stdout)
            # manually set host if we fail, since it won't always be parsable
            # through stdout
            if result.success is False:
                result.host = host
            self.logger.info(f"Got sample: {vars(result)}")
            
            yield self.create_new_result(
                # We use vars here because create_new_result expects
                # dict objects, not dataclasses
                data=vars(result), 
                config={"samples": self.config.samples, "count": self.config.count},
                # tag is a method for labeling results for exporters
                # right now it specifies the ES index to export to
                tag="jupyter"
            )
        
        plural = "s" if self.config.samples > 1 else ""
        self.logger.info(
            f"Finised collecting {self.config.samples} sample{plural} against {host}"
        )

    def collect(self) -> Iterable[BenchmarkResult]:
        """
        Run the Ping Test Benchmark and collect results.
        """
        
        self.logger.info("Running pings and collecting results.")
        self.logger.debug(f"Using config: {vars(self.config.params)}")
        if isinstance(self.config.host, str):
            yield from self.ping_host(self.config.host)
        else:
            for host in self.config.host:
                yield from self.ping_host(host)
        self.logger.info("Finished")

    def cleanup(self) -> bool:
        """
        Cleanup the Ping Test Benchmark.
        
        This method removes the temporary file at ``/tmp/snafu-pingtest`` to let others 
        know that the benchmark has finished running.
        
        Returns
        -------
        bool
            True if the temporary file was deleted successfully, otherwise False.
        """
        
        try:
            os.remove(self.TMP_FILE_PATH)
        except Exception as e:
            self.logger.critical(
                f"Unable to remove temporary file at {self.TMP_FILE_PATH}: {e}",
                exc_info=True
            )
            
            return False
        else:
            self.logger.info(
                f"Successfully removed temp file at {self.TMP_FILE_PATH}"
            )
            return True<jupyter_output><empty_output><jupyter_text>We have finished our new ping test benchmark! Let's try it out! We'll ping three hosts:* `www.google.com`: Depending on the build environment, this domain will show either 100% success (ICMP enabled) or 100% failure (ICMP disabled).* `www.idontexist.heythere`: A domain name which doesn't exist. Ping should exit with a failure saying that the host couldn't be resolved* `localhost`: We'll ping localhost, as we know regardless of environment we'll be able to ping ourselves.<jupyter_code>from snafu.registry import TOOLS
from pprint import pprint
import logging
pingtest = TOOLS["pingtest"]()

# All Benchmark loggers work under the "snafu" logger
logger = logging.getLogger("snafu")
if not logger.hasHandlers():
    logger.addHandler(logging.StreamHandler())
    logger.setLevel(logging.DEBUG)

!rm -rf /tmp/snafu-pingtest
!rm -f my_config.yaml

# Set some config parameters
# Config file
!echo "samples: 1" > my_config.yaml
!echo "count: 5" >> my_config.yaml
# OS ENV
import os
os.environ["HOST"] = "[www.google.com,www.idontexist.heythere,localhost]"

# Parse arguments and print result
# Since we aren't running within the main script (run_snafu.py),
# need to add the config option manually
pingtest.config.parser.add_argument("--config", is_config_file=True)
pingtest.config.parse_args(
    "--config my_config.yaml --labels=notebook=true --uuid 1337 --user snafu "
    "--htlhcdtwy=no".split(" ")
)

# The base benchmark class includes a run method that runs setup -> collect -> cleanup
results = list(pingtest.run())

!rm -rf /tmp/snafu-pingtest
!rm -f my_config.yaml
print(f"Got {len(results)} results")
for result in results[:5]:
    pprint(vars(result))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09003554034487297, 'fraction_numerical': 0.0022377254179281295, 'mean_word_length': 2.192121848739496, 'pattern_counts': {'":': 2, '<': 33, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '93340', 'license': 'Apache-2.0', 'n_tokens_mistral': 8368, 'n_tokens_neox': 8070, 'n_words': 2989, 'path': 'docs/source/contributing/adding_workloads.ipynb', 'repo_name': 'whitleykeith/benchmark-wrapper', 'size': '30388'}