



  

Lab documentation

 [Lab](index.html#document-index) 

 stable

 

Tutorials

* [Lab tutorial](index.html#document-lab.tutorial)

* [Downward Lab tutorial](index.html#document-downward.tutorial)

How-to Guides

* [Frequently asked questions](index.html#document-faq)

* [Parse output](index.html#document-lab.parser)

* [Run other planners](index.html#document-ff)

* [Run Singularity images](index.html#document-singularity)

Reference

* [`lab.experiment` — Create experiments](index.html#document-lab.experiment)

* [`lab.reports` – Make reports](index.html#document-lab.reports)

* [`downward.experiment` — Fast Downward experiment](index.html#document-downward.experiment)

* [`downward.reports` — Fast Downward reports](index.html#document-downward.reports)

General

* [Concepts](index.html#document-lab.concepts)

* [Changelog](index.html#document-news)

[Lab](index.html#document-index)

* [Docs](index.html#document-index) »

* Lab documentation

* [Edit on GitHub](https://github.com/aibasel/lab/blob/b6f80cefe1c3745282cf4432c2989d0a03ddafe3/docs/index.rst)

---

Lab and Downward Lab[¶](#lab-and-downward-lab "Permalink to this headline")

===========================================================================

**Lab** is a Python package for evaluating solvers on benchmark sets.

Experiments can run on a single machine or on a computer cluster. The

package also contains code for parsing results and creating reports.

The **Downward Lab** Python package facilitates running experiments for

the [Fast Downward](http://www.fast-downward.org) planning system. It

uses the generic experimentation package **Lab**. Currently, Lab and

Downward Lab are distributed together.

**Code**: <https://github.com/aibasel/lab>

**Documentation**: <https://lab.readthedocs.io>

**Cite**: please cite Downward Lab by using

```

@Misc{seipp-et-al-zenodo2017,

  author =       "Jendrik Seipp and Florian Pommerening and

                  Silvan Sievers and Malte Helmert",

  title =        "{Downward} {Lab}",

  publisher =    "Zenodo",

  year =         "2017",

  howpublished = "\url{https://doi.org/10.5281/zenodo.790461}"

}

```

Lab tutorial[¶](#lab-tutorial "Permalink to this headline")

-----------------------------------------------------------

Note

During [ICAPS 2020](https://icaps20.icaps-conference.org/), we gave

an [online talk about Lab and Downward Lab](https://icaps20subpages.icaps-conference.org/tutorials/evaluating-planners-with-downward-lab/)

(version 6.2). The first half of the presentation shows how to use Lab

to run experiments for a solver. You can find the recording [here](https://www.youtube.com/watch?v=39tIUsxbh9w).

### Install Lab[¶](#install-lab "Permalink to this headline")

Lab requires Python 3.7+ and Linux (e.g., Ubuntu). We recommend installing

Lab in a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html). This has the advantage

that there are no modifications to the system-wide configuration, and that

you can create multiple environments with different Lab versions (e.g.,

for different papers) without conflicts:

```

# Install required packages, including virtualenv.

sudo apt install python3 python3-venv

# Create a new directory for your experiments.

mkdir experiments-for-my-paper

cd experiments-for-my-paper

# If PYTHONPATH is set, unset it to obtain a clean environment.

unset PYTHONPATH

# Create and activate a Python virtual environment for Lab.

python3 -m venv --prompt my-paper .venv

source .venv/bin/activate

# Install Lab in the virtual environment.

pip install -U pip wheel

pip install lab # or preferably a specific version with lab==x.y

# Store installed packages and exact versions for reproducibility.

# Ignore pkg-resources package (https://github.com/pypa/pip/issues/4022).

pip freeze | grep -v "pkg-resources" > requirements.txt

```

Please note that before running an experiment script you need to

activate the virtual environment with:

```

source .venv/bin/activate

```

We recommend clearing the `PYTHONPATH` variable before activating the

virtual environment.

### Run tutorial experiment[¶](#run-tutorial-experiment "Permalink to this headline")

The following script shows a simple experiment that runs a naive vertex

cover solver on a set of benchmarks.

../examples/vertex-cover/exp.py[¶](#id2 "Permalink to this code")

```

#! /usr/bin/env python

"""

Example experiment using a simple vertex cover solver.

"""

import glob

import os

import platform

from downward.reports.absolute import AbsoluteReport

from lab.environments import BaselSlurmEnvironment, LocalEnvironment

from lab.experiment import Experiment

from lab.reports import Attribute

# Create custom report class with suitable info and error attributes.

class BaseReport(AbsoluteReport):

    INFO\_ATTRIBUTES = ["time\_limit", "memory\_limit", "seed"]

    ERROR\_ATTRIBUTES = [

        "domain",

        "problem",

        "algorithm",

        "unexplained\_errors",

        "error",

        "node",

    ]

NODE = platform.node()

REMOTE = NODE.endswith(".scicore.unibas.ch") or NODE.endswith(".cluster.bc2.ch")

SCRIPT\_DIR = os.path.dirname(os.path.abspath(\_\_file\_\_))

BENCHMARKS\_DIR = os.path.join(SCRIPT\_DIR, "benchmarks")

BHOSLIB\_GRAPHS = sorted(glob.glob(os.path.join(BENCHMARKS\_DIR, "bhoslib", "\*.mis")))

RANDOM\_GRAPHS = sorted(glob.glob(os.path.join(BENCHMARKS\_DIR, "random", "\*.txt")))

ALGORITHMS = ["2approx", "greedy"]

SEED = 2018

TIME\_LIMIT = 1800

MEMORY\_LIMIT = 2048

if REMOTE:

    ENV = BaselSlurmEnvironment(email="my.name@unibas.ch")

    SUITE = BHOSLIB\_GRAPHS + RANDOM\_GRAPHS

else:

    ENV = LocalEnvironment(processes=2)

    # Use smaller suite for local tests.

    SUITE = BHOSLIB\_GRAPHS[:1] + RANDOM\_GRAPHS[:1]

ATTRIBUTES = [

    "cover",

    "cover\_size",

    "error",

    "solve\_time",

    "solver\_exit\_code",

    Attribute("solved", absolute=True),

]

# Create a new experiment.

exp = Experiment(environment=ENV)

# Add solver to experiment and make it available to all runs.

exp.add\_resource("solver", os.path.join(SCRIPT\_DIR, "solver.py"))

# Add custom parser.

exp.add\_parser("parser.py")

for algo in ALGORITHMS:

    for task in SUITE:

        run = exp.add\_run()

        # Create a symbolic link and an alias. This is optional. We

        # could also use absolute paths in add\_command().

        run.add\_resource("task", task, symlink=True)

        run.add\_command(

            "solve",

            ["{solver}", "--seed", str(SEED), "{task}", algo],

            time\_limit=TIME\_LIMIT,

            memory\_limit=MEMORY\_LIMIT,

        )

        # AbsoluteReport needs the following attributes:

        # 'domain', 'problem' and 'algorithm'.

        domain = os.path.basename(os.path.dirname(task))

        task\_name = os.path.basename(task)

        run.set\_property("domain", domain)

        run.set\_property("problem", task\_name)

        run.set\_property("algorithm", algo)

        # BaseReport needs the following properties:

        # 'time\_limit', 'memory\_limit', 'seed'.

        run.set\_property("time\_limit", TIME\_LIMIT)

        run.set\_property("memory\_limit", MEMORY\_LIMIT)

        run.set\_property("seed", SEED)

        # Every run has to have a unique id in the form of a list.

        run.set\_property("id", [algo, domain, task\_name])

# Add step that writes experiment files to disk.

exp.add\_step("build", exp.build)

# Add step that executes all runs.

exp.add\_step("start", exp.start\_runs)

# Add step that collects properties from run directories and

# writes them to \*-eval/properties.

exp.add\_fetcher(name="fetch")

# Make a report.

exp.add\_report(BaseReport(attributes=ATTRIBUTES), outfile="report.html")

# Parse the commandline and run the given steps.

exp.run\_steps()

```

You can see the available steps with

```

./exp.py

```

Select steps by name or index:

```

./exp.py build

./exp.py 2

./exp.py 3 4

```

Here is the parser that the experiment uses:

../examples/vertex-cover/parser.py[¶](#id3 "Permalink to this code")

```

#! /usr/bin/env python

"""

Solver example output:

Algorithm: 2approx

Cover: set([1, 3, 5, 6, 7, 8, 9])

Cover size: 7

Solve time: 0.000771s

"""

from lab.parser import Parser

def solved(content, props):

    props["solved"] = int("cover" in props)

def error(content, props):

    if props["solved"]:

        props["error"] = "cover-found"

    else:

        props["error"] = "unsolved"

if \_\_name\_\_ == "\_\_main\_\_":

    parser = Parser()

    parser.add\_pattern(

        "node", r"node: (.+)\n", type=str, file="driver.log", required=True

    )

    parser.add\_pattern(

        "solver\_exit\_code", r"solve exit code: (.+)\n", type=int, file="driver.log"

    )

    parser.add\_pattern("cover", r"Cover: (\{.\*\})", type=str)

    parser.add\_pattern("cover\_size", r"Cover size: (\d+)\n", type=int)

    parser.add\_pattern("solve\_time", r"Solve time: (.+)s", type=float)

    parser.add\_function(solved)

    parser.add\_function(error)

    parser.parse()

```

Find out how to create your own experiments by browsing the [Lab API](lab.experiment.html).

Downward Lab tutorial[¶](#downward-lab-tutorial "Permalink to this headline")

-----------------------------------------------------------------------------

This tutorial shows you how to install Downward Lab and how to create a

simple experiment for Fast Downward that compares two heuristics, the

causal graph (CG) heuristic and the FF heuristic. There are many ways for

setting up your experiments. This tutorial gives you an opinionated

alternative that has proven to work well in practice.

Note

During [ICAPS 2020](https://icaps20.icaps-conference.org/), we gave

an online [Downward Lab presentation](https://icaps20subpages.icaps-conference.org/tutorials/evaluating-planners-with-downward-lab/)

(version 6.2). The second half of the presentation covers this

tutorial and you can find the recording [here](https://www.youtube.com/watch?v=39tIUsxbh9w).

### Installation[¶](#installation "Permalink to this headline")

Lab requires **Python 3.7+** and **Linux**. To run Fast Downward

experiments, you’ll need a **Fast Downward** repository, planning

**benchmarks** and a plan **validator**.

```

# Install required packages.

sudo apt install bison cmake flex g++ git make python3 python3-venv

# Create directory for holding binaries and scripts.

mkdir --parents ~/bin

# Make directory for all projects related to Fast Downward.

mkdir downward-projects

cd downward-projects

# Install the plan validator VAL.

git clone https://github.com/KCL-Planning/VAL.git

cd VAL

# Newer VAL versions need time stamps, so we use an old version

# (https://github.com/KCL-Planning/VAL/issues/46).

git checkout a556539

make clean # Remove old binaries.

sed -i 's/-Werror //g' Makefile # Ignore warnings.

make

cp validate ~/bin/ # Add binary to a directory on your ``$PATH``.

# Return to projects directory.

cd ../

# Download planning tasks.

git clone https://github.com/aibasel/downward-benchmarks.git benchmarks

# Clone Fast Downward and let it solve an example task.

git clone https://github.com/aibasel/downward.git

cd downward

./build.py

./fast-downward.py ../benchmarks/grid/prob01.pddl --search "astar(lmcut())"

```

If Fast Downward doesn’t compile, see

<http://www.fast-downward.org/ObtainingAndRunningFastDownward> and

<http://www.fast-downward.org/LPBuildInstructions>. We now create a new

directory for our CG-vs-FF project. By putting it into the Fast Downward

repo under `experiments/`, it’s easy to share both the code and

experiment scripts with your collaborators.

```

# Create new branch.

git checkout -b cg-vs-ff main

# Create a new directory for your experiments in Fast Downward repo.

cd experiments

mkdir cg-vs-ff

cd cg-vs-ff

```

Now it’s time to install Lab. We install it in a [Python virtual

environment](https://docs.python.org/3/tutorial/venv.html) specific to

the cg-vs-ff project. This has the advantage that there are no

modifications to the system-wide configuration, and that you can have

multiple projects with different Lab versions (e.g., for different

papers).

```

# Create and activate a Python 3 virtual environment for Lab.

python3 -m venv --prompt cg-vs-ff .venv

source .venv/bin/activate

# Install Lab in the virtual environment.

pip install -U pip wheel # It's good to have new versions of these.

pip install lab # or preferably a specific version with lab==x.y

# Store installed packages and exact versions for reproducibility.

# Ignore pkg-resources package (https://github.com/pypa/pip/issues/4022).

pip freeze | grep -v "pkg-resources" > requirements.txt

git add requirements.txt

git commit -m "Store requirements for experiments."

```

To use the same versions of your requirements on a different computer, use

`pip install -r requirements.txt` instead of the `pip install`

commands above.

Add to your `~/.bashrc` file:

```

# Make executables in ~/bin directory available globally.

export PATH="${PATH}:${HOME}/bin"

# Some example experiments need these two environment variables.

export DOWNWARD\_BENCHMARKS=/path/to/downward-projects/benchmarks # Adapt path

export DOWNWARD\_REPO=/path/to/downward-projects/downward # Adapt path

```

Add to your `~/.bash\_aliases` file:

```

# Activate virtualenv and unset PYTHONPATH to obtain isolated virtual environments.

alias venv="unset PYTHONPATH; source .venv/bin/activate"

```

Finally, reload `.bashrc` (which usually also reloads `~/.bash\_aliases`):

```

source ~/.bashrc

```

You can activate virtual environments now by running `venv` in

directories containing a `.venv` subdirectory.

### Run tutorial experiment[¶](#run-tutorial-experiment "Permalink to this headline")

The files below are two experiment scripts, a `project.py` module that bundles

common functionality for all experiments related to the project, a parser

script, and a script for collecting results and making reports. You can use the

files as a basis for your own experiments. They are available in the [Lab repo](https://github.com/aibasel/lab/tree/main/examples/downward). Copy the files

into `experiments/my-exp-dir`.

Make sure the experiment script and the parser are executable. Then you

can see the available steps with

```

./2020-09-11-A-cg-vs-ff.py

```

Run all steps with

```

./2020-09-11-A-cg-vs-ff.py --all

```

Run individual steps with

```

./2020-09-11-A-cg-vs-ff.py build

./2020-09-11-A-cg-vs-ff.py 2

./2020-09-11-A-cg-vs-ff.py 3 6 7

```

The `2020-09-11-A-cg-vs-ff.py` script uses the

[`downward.experiment.FastDownwardExperiment`](index.html#downward.experiment.FastDownwardExperiment "downward.experiment.FastDownwardExperiment") class, which reduces the

amount of code you need to write, but assumes a rigid structure of the

experiment: it only allows you to run each added algorithm on each added task,

and individual runs cannot be customized. If you need more flexibility, you can

employ the [`lab.experiment.Experiment`](index.html#lab.experiment.Experiment "lab.experiment.Experiment") class instead and fill it by using

[`FastDownwardAlgorithm`](index.html#downward.experiment.FastDownwardAlgorithm "downward.experiment.FastDownwardAlgorithm"),

[`FastDownwardRun`](index.html#downward.experiment.FastDownwardRun "downward.experiment.FastDownwardRun"),

[`CachedFastDownwardRevision`](index.html#downward.cached_revision.CachedFastDownwardRevision "downward.cached_revision.CachedFastDownwardRevision"), and [`Task`](index.html#downward.suites.Task "downward.suites.Task") objects. The `2020-09-11-A-cg-vs-ff.py` script shows

an example. See the [Downward Lab API](downward.experiment.html) for a

reference on all Downward Lab classes.

../examples/downward/2020-09-11-A-cg-vs-ff.py[¶](#id2 "Permalink to this code")

```

#! /usr/bin/env python

import os

import shutil

import project

REPO = project.get\_repo\_base()

BENCHMARKS\_DIR = os.environ["DOWNWARD\_BENCHMARKS"]

SCP\_LOGIN = "myname@myserver.com"

REMOTE\_REPOS\_DIR = "/infai/seipp/projects"

# If REVISION\_CACHE is None, the default "./data/revision-cache/" is used.

REVISION\_CACHE = os.environ.get("DOWNWARD\_REVISION\_CACHE")

if project.REMOTE:

    SUITE = project.SUITE\_SATISFICING

    ENV = project.BaselSlurmEnvironment(email="my.name@myhost.ch")

else:

    SUITE = ["depot:p01.pddl", "grid:prob01.pddl", "gripper:prob01.pddl"]

    ENV = project.LocalEnvironment(processes=2)

CONFIGS = [

    (f"{index:02d}-{h\_nick}", ["--search", f"eager\_greedy([{h}])"])

    for index, (h\_nick, h) in enumerate(

        [

            ("cg", "cg(transform=adapt\_costs(one))"),

            ("ff", "ff(transform=adapt\_costs(one))"),

        ],

        start=1,

    )

]

BUILD\_OPTIONS = []

DRIVER\_OPTIONS = ["--overall-time-limit", "5m"]

REV\_NICKS = [

    ("main", ""),

]

ATTRIBUTES = [

    "error",

    "run\_dir",

    "search\_start\_time",

    "search\_start\_memory",

    "total\_time",

    "h\_values",

    "coverage",

    "expansions",

    "memory",

    project.EVALUATIONS\_PER\_TIME,

]

exp = project.FastDownwardExperiment(environment=ENV, revision\_cache=REVISION\_CACHE)

for config\_nick, config in CONFIGS:

    for rev, rev\_nick in REV\_NICKS:

        algo\_name = f"{rev\_nick}:{config\_nick}" if rev\_nick else config\_nick

        exp.add\_algorithm(

            algo\_name,

            REPO,

            rev,

            config,

            build\_options=BUILD\_OPTIONS,

            driver\_options=DRIVER\_OPTIONS,

        )

exp.add\_suite(BENCHMARKS\_DIR, SUITE)

exp.add\_parser(exp.EXITCODE\_PARSER)

exp.add\_parser(exp.TRANSLATOR\_PARSER)

exp.add\_parser(exp.SINGLE\_SEARCH\_PARSER)

exp.add\_parser(project.DIR / "parser.py")

exp.add\_parser(exp.PLANNER\_PARSER)

exp.add\_step("build", exp.build)

exp.add\_step("start", exp.start\_runs)

exp.add\_fetcher(name="fetch")

if not project.REMOTE:

    exp.add\_step("remove-eval-dir", shutil.rmtree, exp.eval\_dir, ignore\_errors=True)

    project.add\_scp\_step(exp, SCP\_LOGIN, REMOTE\_REPOS\_DIR)

project.add\_absolute\_report(

    exp, attributes=ATTRIBUTES, filter=[project.add\_evaluations\_per\_time]

)

attributes = ["expansions"]

pairs = [

    ("01-cg", "02-ff"),

]

suffix = "-rel" if project.RELATIVE else ""

for algo1, algo2 in pairs:

    for attr in attributes:

        exp.add\_report(

            project.ScatterPlotReport(

                relative=project.RELATIVE,

                get\_category=None if project.TEX else lambda run1, run2: run1["domain"],

                attributes=[attr],

                filter\_algorithm=[algo1, algo2],

                filter=[project.add\_evaluations\_per\_time],

                format="tex" if project.TEX else "png",

            ),

            name=f"{exp.name}-{algo1}-vs-{algo2}-{attr}{suffix}",

        )

exp.run\_steps()

```

../examples/downward/2020-09-11-B-bounded-cost.py[¶](#id3 "Permalink to this code")

```

#! /usr/bin/env python

import json

import os

import shutil

from downward import suites

from downward.cached\_revision import CachedFastDownwardRevision

from downward.experiment import FastDownwardAlgorithm, FastDownwardRun

from lab.experiment import Experiment

import project

REPO = project.get\_repo\_base()

BENCHMARKS\_DIR = os.environ["DOWNWARD\_BENCHMARKS"]

SCP\_LOGIN = "myname@myserver.com"

REMOTE\_REPOS\_DIR = "/infai/seipp/projects"

BOUNDS\_FILE = "bounds.json"

SUITE = ["depot:p01.pddl", "grid:prob01.pddl", "gripper:prob01.pddl"]

# If REVISION\_CACHE is None, the default "./data/revision-cache/" is used.

REVISION\_CACHE = os.environ.get("DOWNWARD\_REVISION\_CACHE")

if project.REMOTE:

    # ENV = project.BaselSlurmEnvironment(email="my.name@myhost.ch")

    ENV = project.TetralithEnvironment(

        email="first.last@liu.se", extra\_options="#SBATCH --account=snic2022-5-341"

    )

    SUITE = project.SUITE\_OPTIMAL\_STRIPS

else:

    ENV = project.LocalEnvironment(processes=2)

CONFIGS = [

    ("ff", ["--search", "lazy\_greedy([ff()], bound=BOUND)"]),

]

BUILD\_OPTIONS = []

DRIVER\_OPTIONS = [

    "--validate",

    "--overall-time-limit",

    "5m",

    "--overall-memory-limit",

    "3584M",

]

# Pairs of revision identifier and optional revision nick.

REV\_NICKS = [

    ("main", ""),

]

ATTRIBUTES = [

    "error",

    "run\_dir",

    "search\_start\_time",

    "search\_start\_memory",

    "total\_time",

    "h\_values",

    "coverage",

    "expansions",

    "memory",

    project.EVALUATIONS\_PER\_TIME,

]

exp = Experiment(environment=ENV)

for rev, rev\_nick in REV\_NICKS:

    cached\_rev = CachedFastDownwardRevision(REVISION\_CACHE, REPO, rev, BUILD\_OPTIONS)

    cached\_rev.cache()

    exp.add\_resource("", cached\_rev.path, cached\_rev.get\_relative\_exp\_path())

    for config\_nick, config in CONFIGS:

        algo\_name = f"{rev\_nick}-{config\_nick}" if rev\_nick else config\_nick

        bounds = {}

        with open(BOUNDS\_FILE) as f:

            bounds = json.load(f)

        for task in suites.build\_suite(BENCHMARKS\_DIR, SUITE):

            upper\_bound = bounds[f"{task.domain}:{task.problem}"]

            if upper\_bound is None:

                upper\_bound = "infinity"

            config\_with\_bound = config.copy()

            config\_with\_bound[-1] = config\_with\_bound[-1].replace(

                "bound=BOUND", f"bound={upper\_bound}"

            )

            algo = FastDownwardAlgorithm(

                algo\_name,

                cached\_rev,

                DRIVER\_OPTIONS,

                config\_with\_bound,

            )

            run = FastDownwardRun(exp, algo, task)

            exp.add\_run(run)

exp.add\_parser(project.FastDownwardExperiment.EXITCODE\_PARSER)

exp.add\_parser(project.FastDownwardExperiment.TRANSLATOR\_PARSER)

exp.add\_parser(project.FastDownwardExperiment.SINGLE\_SEARCH\_PARSER)

exp.add\_parser(project.DIR / "parser.py")

exp.add\_parser(project.FastDownwardExperiment.PLANNER\_PARSER)

exp.add\_step("build", exp.build)

exp.add\_step("start", exp.start\_runs)

exp.add\_fetcher(name="fetch")

if not project.REMOTE:

    exp.add\_step("remove-eval-dir", shutil.rmtree, exp.eval\_dir, ignore\_errors=True)

    project.add\_scp\_step(exp, SCP\_LOGIN, REMOTE\_REPOS\_DIR)

project.add\_absolute\_report(

    exp,

    attributes=ATTRIBUTES,

    filter=[project.add\_evaluations\_per\_time, project.group\_domains],

)

exp.run\_steps()

```

../examples/downward/project.py[¶](#id4 "Permalink to this code")

```

from pathlib import Path

import platform

import re

import subprocess

import sys

from downward.experiment import FastDownwardExperiment

from downward.reports.absolute import AbsoluteReport

from downward.reports.scatter import ScatterPlotReport

from downward.reports.taskwise import TaskwiseReport

from lab.environments import (

    BaselSlurmEnvironment,

    LocalEnvironment,

    TetralithEnvironment,

)

from lab.experiment import ARGPARSER

from lab.reports import Attribute, geometric\_mean

# Silence import-unused messages. Experiment scripts may use these imports.

assert (

    BaselSlurmEnvironment

    and FastDownwardExperiment

    and LocalEnvironment

    and ScatterPlotReport

    and TaskwiseReport

    and TetralithEnvironment

)

DIR = Path(\_\_file\_\_).resolve().parent

SCRIPT = Path(sys.argv[0]).resolve()

NODE = platform.node()

# Cover both the Basel and Linköping clusters for simplicity.

REMOTE = NODE.endswith((".scicore.unibas.ch", ".cluster.bc2.ch")) or re.match(

    r"tetralith\d+\.nsc\.liu\.se|n\d+", NODE

)

def parse\_args():

    ARGPARSER.add\_argument("--tex", action="store\_true", help="produce LaTeX output")

    ARGPARSER.add\_argument(

        "--relative", action="store\_true", help="make relative scatter plots"

    )

    return ARGPARSER.parse\_args()

ARGS = parse\_args()

TEX = ARGS.tex

RELATIVE = ARGS.relative

EVALUATIONS\_PER\_TIME = Attribute(

    "evaluations\_per\_time", min\_wins=False, function=geometric\_mean, digits=1

)

# Generated by "./suites.py satisficing" in aibasel/downward-benchmarks repo.

# fmt: off

SUITE\_SATISFICING = [

    "agricola-sat18-strips", "airport", "assembly", "barman-sat11-strips",

    "barman-sat14-strips", "blocks", "caldera-sat18-adl",

    "caldera-split-sat18-adl", "cavediving-14-adl", "childsnack-sat14-strips",

    "citycar-sat14-adl", "data-network-sat18-strips", "depot", "driverlog",

    "elevators-sat08-strips", "elevators-sat11-strips", "flashfill-sat18-adl",

    "floortile-sat11-strips", "floortile-sat14-strips", "freecell",

    "ged-sat14-strips", "grid", "gripper", "hiking-sat14-strips",

    "logistics00", "logistics98", "maintenance-sat14-adl", "miconic",

    "miconic-fulladl", "miconic-simpleadl", "movie", "mprime", "mystery",

    "nomystery-sat11-strips", "nurikabe-sat18-adl", "openstacks",

    "openstacks-sat08-adl", "openstacks-sat08-strips",

    "openstacks-sat11-strips", "openstacks-sat14-strips", "openstacks-strips",

    "optical-telegraphs", "organic-synthesis-sat18-strips",

    "organic-synthesis-split-sat18-strips", "parcprinter-08-strips",

    "parcprinter-sat11-strips", "parking-sat11-strips", "parking-sat14-strips",

    "pathways", "pegsol-08-strips", "pegsol-sat11-strips", "philosophers",

    "pipesworld-notankage", "pipesworld-tankage", "psr-large", "psr-middle",

    "psr-small", "rovers", "satellite", "scanalyzer-08-strips",

    "scanalyzer-sat11-strips", "schedule", "settlers-sat18-adl",

    "snake-sat18-strips", "sokoban-sat08-strips", "sokoban-sat11-strips",

    "spider-sat18-strips", "storage", "termes-sat18-strips",

    "tetris-sat14-strips", "thoughtful-sat14-strips", "tidybot-sat11-strips",

    "tpp", "transport-sat08-strips", "transport-sat11-strips",

    "transport-sat14-strips", "trucks", "trucks-strips",

    "visitall-sat11-strips", "visitall-sat14-strips",

    "woodworking-sat08-strips", "woodworking-sat11-strips", "zenotravel",

]

SUITE\_OPTIMAL\_STRIPS = [

    "agricola-opt18-strips", "airport", "barman-opt11-strips",

    "barman-opt14-strips", "blocks", "childsnack-opt14-strips",

    "data-network-opt18-strips", "depot", "driverlog", "elevators-opt08-strips",

    "elevators-opt11-strips", "floortile-opt11-strips", "floortile-opt14-strips",

    "freecell", "ged-opt14-strips", "grid", "gripper", "hiking-opt14-strips",

    "logistics00", "logistics98", "miconic", "movie", "mprime", "mystery",

    "nomystery-opt11-strips", "openstacks-opt08-strips", "openstacks-opt11-strips",

    "openstacks-opt14-strips", "openstacks-strips", "organic-synthesis-opt18-strips",

    "organic-synthesis-split-opt18-strips", "parcprinter-08-strips",

    "parcprinter-opt11-strips", "parking-opt11-strips", "parking-opt14-strips",

    "pathways", "pegsol-08-strips", "pegsol-opt11-strips",

    "petri-net-alignment-opt18-strips", "pipesworld-notankage", "pipesworld-tankage",

    "psr-small", "rovers", "satellite", "scanalyzer-08-strips",

    "scanalyzer-opt11-strips", "snake-opt18-strips", "sokoban-opt08-strips",

    "sokoban-opt11-strips", "spider-opt18-strips", "storage", "termes-opt18-strips",

    "tetris-opt14-strips", "tidybot-opt11-strips", "tidybot-opt14-strips", "tpp",

    "transport-opt08-strips", "transport-opt11-strips", "transport-opt14-strips",

    "trucks-strips", "visitall-opt11-strips", "visitall-opt14-strips",

    "woodworking-opt08-strips", "woodworking-opt11-strips", "zenotravel",

]

DOMAIN\_GROUPS = {

    "airport": ["airport"],

    "assembly": ["assembly"],

    "barman": [

        "barman", "barman-opt11-strips", "barman-opt14-strips",

        "barman-sat11-strips", "barman-sat14-strips"],

    "blocksworld": ["blocks", "blocksworld"],

    "cavediving": ["cavediving-14-adl"],

    "childsnack": ["childsnack-opt14-strips", "childsnack-sat14-strips"],

    "citycar": ["citycar-opt14-adl", "citycar-sat14-adl"],

    "depots": ["depot", "depots"],

    "driverlog": ["driverlog"],

    "elevators": [

        "elevators-opt08-strips", "elevators-opt11-strips",

        "elevators-sat08-strips", "elevators-sat11-strips"],

    "floortile": [

        "floortile-opt11-strips", "floortile-opt14-strips",

        "floortile-sat11-strips", "floortile-sat14-strips"],

    "freecell": ["freecell"],

    "ged": ["ged-opt14-strips", "ged-sat14-strips"],

    "grid": ["grid"],

    "gripper": ["gripper"],

    "hiking": ["hiking-opt14-strips", "hiking-sat14-strips"],

    "logistics": ["logistics98", "logistics00"],

    "maintenance": ["maintenance-opt14-adl", "maintenance-sat14-adl"],

    "miconic": ["miconic", "miconic-strips"],

    "miconic-fulladl": ["miconic-fulladl"],

    "miconic-simpleadl": ["miconic-simpleadl"],

    "movie": ["movie"],

    "mprime": ["mprime"],

    "mystery": ["mystery"],

    "nomystery": ["nomystery-opt11-strips", "nomystery-sat11-strips"],

    "openstacks": [

        "openstacks", "openstacks-strips", "openstacks-opt08-strips",

        "openstacks-opt11-strips", "openstacks-opt14-strips",

        "openstacks-sat08-adl", "openstacks-sat08-strips",

        "openstacks-sat11-strips", "openstacks-sat14-strips",

        "openstacks-opt08-adl", "openstacks-sat08-adl"],

    "optical-telegraphs": ["optical-telegraphs"],

    "parcprinter": [

        "parcprinter-08-strips", "parcprinter-opt11-strips", "parcprinter-sat11-strips"],

    "parking": [

        "parking-opt11-strips", "parking-opt14-strips",

        "parking-sat11-strips", "parking-sat14-strips"],

    "pathways": ["pathways"],

    "pathways-noneg": ["pathways-noneg"],

    "pegsol": ["pegsol-08-strips", "pegsol-opt11-strips", "pegsol-sat11-strips"],

    "philosophers": ["philosophers"],

    "pipes-nt": ["pipesworld-notankage"],

    "pipes-t": ["pipesworld-tankage"],

    "psr": ["psr-middle", "psr-large", "psr-small"],

    "rovers": ["rover", "rovers"],

    "satellite": ["satellite"],

    "scanalyzer": [

        "scanalyzer-08-strips", "scanalyzer-opt11-strips", "scanalyzer-sat11-strips"],

    "schedule": ["schedule"],

    "sokoban": [

        "sokoban-opt08-strips", "sokoban-opt11-strips",

        "sokoban-sat08-strips", "sokoban-sat11-strips"],

    "storage": ["storage"],

    "tetris": ["tetris-opt14-strips", "tetris-sat14-strips"],

    "thoughtful": ["thoughtful-sat14-strips"],

    "tidybot": [

        "tidybot-opt11-strips", "tidybot-opt14-strips",

        "tidybot-sat11-strips", "tidybot-sat14-strips"],

    "tpp": ["tpp"],

    "transport": [

        "transport-opt08-strips", "transport-opt11-strips", "transport-opt14-strips",

        "transport-sat08-strips", "transport-sat11-strips", "transport-sat14-strips"],

    "trucks": ["trucks", "trucks-strips"],

    "visitall": [

        "visitall-opt11-strips", "visitall-opt14-strips",

        "visitall-sat11-strips", "visitall-sat14-strips"],

    "woodworking": [

        "woodworking-opt08-strips", "woodworking-opt11-strips",

        "woodworking-sat08-strips", "woodworking-sat11-strips"],

    "zenotravel": ["zenotravel"],

    # IPC 2018:

    "agricola": ["agricola", "agricola-opt18-strips", "agricola-sat18-strips"],

    "caldera": ["caldera-opt18-adl", "caldera-sat18-adl"],

    "caldera-split": ["caldera-split-opt18-adl", "caldera-split-sat18-adl"],

    "data-network": [

        "data-network", "data-network-opt18-strips", "data-network-sat18-strips"],

    "flashfill": ["flashfill-sat18-adl"],

    "nurikabe": ["nurikabe-opt18-adl", "nurikabe-sat18-adl"],

    "organic-split": [

        "organic-synthesis-split", "organic-synthesis-split-opt18-strips",

        "organic-synthesis-split-sat18-strips"],

    "organic" : [

        "organic-synthesis", "organic-synthesis-opt18-strips",

        "organic-synthesis-sat18-strips"],

    "petri-net": [

        "petri-net-alignment", "petri-net-alignment-opt18-strips",

        "petri-net-alignment-sat18-strips"],

    "settlers": ["settlers-opt18-adl", "settlers-sat18-adl"],

    "snake": ["snake", "snake-opt18-strips", "snake-sat18-strips"],

    "spider": ["spider", "spider-opt18-strips", "spider-sat18-strips"],

    "termes": ["termes", "termes-opt18-strips", "termes-sat18-strips"],

}

# fmt: on

DOMAIN\_RENAMINGS = {}

for group\_name, domains in DOMAIN\_GROUPS.items():

    for domain in domains:

        DOMAIN\_RENAMINGS[domain] = group\_name

for group\_name in DOMAIN\_GROUPS:

    DOMAIN\_RENAMINGS[group\_name] = group\_name

def group\_domains(run):

    old\_domain = run["domain"]

    run["domain"] = DOMAIN\_RENAMINGS[old\_domain]

    run["problem"] = old\_domain + "-" + run["problem"]

    run["id"][2] = run["problem"]

    return run

def get\_repo\_base() -> Path:

 """Get base directory of the repository, as an absolute path.

 Search upwards in the directory tree from the main script until a

 directory with a subdirectory named ".git" is found.

 Abort if the repo base cannot be found."""

    path = Path(SCRIPT)

    while path.parent != path:

        if (path / ".git").is\_dir():

            return path

        path = path.parent

    sys.exit("repo base could not be found")

def remove\_file(path: Path):

    try:

        path.unlink()

    except FileNotFoundError:

        pass

def add\_evaluations\_per\_time(run):

    evaluations = run.get("evaluations")

    time = run.get("search\_time")

    if evaluations is not None and evaluations >= 100 and time:

        run["evaluations\_per\_time"] = evaluations / time

    return run

def \_get\_exp\_dir\_relative\_to\_repo():

    repo\_name = get\_repo\_base().name

    script = Path(SCRIPT)

    script\_dir = script.parent

    rel\_script\_dir = script\_dir.relative\_to(get\_repo\_base())

    expname = script.stem

    return repo\_name / rel\_script\_dir / "data" / expname

def add\_scp\_step(exp, login, repos\_dir):

    remote\_exp = Path(repos\_dir) / \_get\_exp\_dir\_relative\_to\_repo()

    exp.add\_step(

        "scp-eval-dir",

        subprocess.call,

        [

            "scp",

            "-r",  # Copy recursively.

            "-C",  # Compress files.

            f"{login}:{remote\_exp}-eval",

            f"{exp.path}-eval",

        ],

    )

def fetch\_algorithm(exp, expname, algo, \*, new\_algo=None):

 """Fetch (and possibly rename) a single algorithm from \*expname\*."""

    new\_algo = new\_algo or algo

    def rename\_and\_filter(run):

        if run["algorithm"] == algo:

            run["algorithm"] = new\_algo

            run["id"][0] = new\_algo

            return run

        return False

    exp.add\_fetcher(

        f"data/{expname}-eval",

        filter=rename\_and\_filter,

        name=f"fetch-{new\_algo}-from-{expname}",

        merge=True,

    )

def fetch\_algorithms(exp, expname, \*, algos=None, name=None, filters=None):

 """

 Fetch multiple or all algorithms.

 """

    assert not expname.rstrip("/").endswith("-eval")

    algos = set(algos or [])

    filters = filters or []

    if algos:

        def algo\_filter(run):

            return run["algorithm"] in algos

        filters.append(algo\_filter)

    exp.add\_fetcher(

        f"data/{expname}-eval",

        filter=filters,

        name=name or f"fetch-from-{expname}",

        merge=True,

    )

def add\_absolute\_report(exp, \*, name=None, outfile=None, \*\*kwargs):

    report = AbsoluteReport(\*\*kwargs)

    if name and not outfile:

        outfile = f"{name}.{report.output\_format}"

    elif outfile and not name:

        name = Path(outfile).name

    elif not name and not outfile:

        name = f"{exp.name}-abs"

        outfile = f"{name}.{report.output\_format}"

    if not Path(outfile).is\_absolute():

        outfile = Path(exp.eval\_dir) / outfile

    exp.add\_report(report, name=name, outfile=outfile)

    if not REMOTE:

        exp.add\_step(f"open-{name}", subprocess.call, ["xdg-open", outfile])

    exp.add\_step(f"publish-{name}", subprocess.call, ["publish", outfile])

```

../examples/downward/parser.py[¶](#id5 "Permalink to this code")

```

#! /usr/bin/env python

import logging

import re

from lab.parser import Parser

class CommonParser(Parser):

    def add\_repeated\_pattern(

        self, name, regex, file="run.log", required=False, type=int

    ):

        def find\_all\_occurences(content, props):

            matches = re.findall(regex, content)

            if required and not matches:

                logging.error(f"Pattern {regex} not found in file {file}")

            props[name] = [type(m) for m in matches]

        self.add\_function(find\_all\_occurences, file=file)

    def add\_bottom\_up\_pattern(

        self, name, regex, file="run.log", required=False, type=int

    ):

        def search\_from\_bottom(content, props):

            reversed\_content = "\n".join(reversed(content.splitlines()))

            match = re.search(regex, reversed\_content)

            if required and not match:

                logging.error(f"Pattern {regex} not found in file {file}")

            if match:

                props[name] = type(match.group(1))

        self.add\_function(search\_from\_bottom, file=file)

def main():

    parser = CommonParser()

    parser.add\_bottom\_up\_pattern(

        "search\_start\_time",

        r"\[t=(.+)s, \d+ KB\] g=0, 1 evaluated, 0 expanded",

        type=float,

    )

    parser.add\_bottom\_up\_pattern(

        "search\_start\_memory",

        r"\[t=.+s, (\d+) KB\] g=0, 1 evaluated, 0 expanded",

        type=int,

    )

    parser.add\_pattern(

        "initial\_h\_value",

        r"f = (\d+) \[1 evaluated, 0 expanded, t=.+s, \d+ KB\]",

        type=int,

    )

    parser.add\_repeated\_pattern(

        "h\_values",

        r"New best heuristic value for .+: (\d+)\n",

        type=int,

    )

    parser.parse()

if \_\_name\_\_ == "\_\_main\_\_":

    main()

```

../examples/downward/01-evaluation.py[¶](#id6 "Permalink to this code")

```

#! /usr/bin/env python

from pathlib import Path

from lab.experiment import Experiment

import project

ATTRIBUTES = [

    "error",

    "run\_dir",

    "planner\_time",

    "initial\_h\_value",

    "coverage",

    "cost",

    "evaluations",

    "memory",

    project.EVALUATIONS\_PER\_TIME,

]

exp = Experiment()

exp.add\_step(

    "remove-combined-properties", project.remove\_file, Path(exp.eval\_dir) / "properties"

)

project.fetch\_algorithm(exp, "2020-09-11-A-cg-vs-ff", "01-cg", new\_algo="cg")

project.fetch\_algorithms(exp, "2020-09-11-B-bounded-cost")

filters = [project.add\_evaluations\_per\_time]

project.add\_absolute\_report(

    exp, attributes=ATTRIBUTES, filter=filters, name=f"{exp.name}"

)

exp.run\_steps()

```

The [Downward Lab API](downward.experiment.html) shows you how to adjust

this example to your needs. You may also find the [other example

experiments](https://github.com/aibasel/lab/tree/main/examples/)

useful.

Frequently asked questions[¶](#frequently-asked-questions "Permalink to this headline")

---------------------------------------------------------------------------------------

### How can I parse and report my own attributes?[¶](#how-can-i-parse-and-report-my-own-attributes "Permalink to this headline")

See the [parsing documentation](lab.parser.html).

### How can I combine the results from multiple experiments?[¶](#how-can-i-combine-the-results-from-multiple-experiments "Permalink to this headline")

```

exp = Experiment('/new/path/to/combined-results')

exp.add\_fetcher('path/to/first/eval/dir')

exp.add\_fetcher('path/to/second/eval/dir')

exp.add\_fetcher('path/to/experiment/dir')

exp.add\_report(AbsoluteReport())

```

### Some runs failed. How can I rerun them?[¶](#some-runs-failed-how-can-i-rerun-them "Permalink to this headline")

If the failed runs were never started, for example, due to grid node

failures, you can simply run the “start” experiment step again. It will

skip all runs that have already been started. Afterwards, run “fetch” and

make reports as usual.

Lab detects which runs have already been started by checking if the

`driver.log` file exists. So if you have failed runs that were already

started, but you want to rerun them anyway, go to their run directories,

remove the `driver.log` files and then run the “start” experiment step

again as above.

### I forgot to parse something. How can I run only the parsers again?[¶](#i-forgot-to-parse-something-how-can-i-run-only-the-parsers-again "Permalink to this headline")

See the [parsing documentation](lab.parser.html) for how to write

parsers. Once you have fixed your existing parsers or added new parsers,

add `exp.add\_parse\_again\_step()` to your experiment script

`my-exp.py` and then call

```

./my-exp.py parse-again

```

### How can I compute a new attribute from multiple runs?[¶](#how-can-i-compute-a-new-attribute-from-multiple-runs "Permalink to this headline")

Consider for example the IPC quality score. It is often computed over the

list of runs for each task. Since filters only work on individual runs, we

can’t compute the score with a single filter, but it is possible by using

two filters as shown below: *store\_costs* saves the list of costs per task

in a dictionary whereas *add\_quality* uses the stored costs to compute IPC

quality scores and adds them to the runs.

```

class QualityFilters:

 """Compute the IPC quality score.

 >>> from downward.reports.absolute import AbsoluteReport

 >>> filters = QualityFilters()

 >>> report = AbsoluteReport(filter=[filters.store\_costs, filters.add\_quality])

 """

    def \_\_init\_\_(self):

        self.tasks\_to\_costs = defaultdict(list)

    def \_get\_task(self, run):

        return (run["domain"], run["problem"])

    def \_compute\_quality(self, cost, all\_costs):

        if cost is None:

            return 0.0

        assert all\_costs

        min\_cost = min(all\_costs)

        if cost == 0:

            assert min\_cost == 0

            return 1.0

        return min\_cost / cost

    def store\_costs(self, run):

        cost = run.get("cost")

        if cost is not None:

            assert run["coverage"]

            self.tasks\_to\_costs[self.\_get\_task(run)].append(cost)

        return True

    def add\_quality(self, run):

        run["quality"] = self.\_compute\_quality(

            run.get("cost"), self.tasks\_to\_costs[self.\_get\_task(run)]

        )

        return run

```

### How can I make reports and plots for results obtained without Lab?[¶](#how-can-i-make-reports-and-plots-for-results-obtained-without-lab "Permalink to this headline")

See [report-external-results.py](https://github.com/aibasel/lab/blob/main/examples/report-external-results.py)

for an example.

### Which experiment class should I use for which Fast Downward revision?[¶](#which-experiment-class-should-i-use-for-which-fast-downward-revision "Permalink to this headline")

* Before CMake: use DownwardExperiment in Lab 1.x

* With CMake and optional validation: use FastDownwardExperiment in Lab 1.x

* With CMake and mandatory validation: use FastDownwardExperiment in Lab 2.x

* New translator exit codes (issue739): use FastDownwardExperiment in Lab >= 3.x

### How can I contribute to Lab?[¶](#how-can-i-contribute-to-lab "Permalink to this headline")

If you’d like to contribute a feature or a bugfix to Lab or Downward Lab,

please see [CONTRIBUTING.md](https://github.com/aibasel/lab/blob/main/CONTRIBUTING.md).

### How can I customize Lab?[¶](#how-can-i-customize-lab "Permalink to this headline")

Lab tries to be easily customizable. That means that you shouldn’t have to

make any changes to the Lab code itself, but rather you should be able to

inherit from Lab classes and implement custom behaviour in your

subclasses. If this doesn’t work in your case, let’s discuss how we can

improve things in a [GitHub issue](https://github.com/aibasel/lab/issues).

That said, it can sometimes be easiest to quickly patch Lab. In this case,

or when you want to run the latest Lab development version, you can clone

the Lab repo and install it (preferable in a virtual environment):

```

git clone https://github.com/aibasel/lab.git /path/to/lab

pip install --editable /path/to/lab

```

The `--editable` flag installs the project in “editable mode”, which

makes any changes under `/path/to/lab` appear immediately in the virtual

environment.

### Which best practices do you recommend for working with Lab?[¶](#which-best-practices-do-you-recommend-for-working-with-lab "Permalink to this headline")

* automate as much as possible but not too much

* use fixed solver revisions (“3a27ea77f” instead of “main”)

* use Python virtual environments

* pin versions of all Python dependencies in `requirements.txt`

* collect common experiment code in project module

* copy experiment scripts for new experiments, don’t change them

* make evaluation locally rather than on remote cluster

* collect exploratory results from multiple experiments

* rerun experiments for camera-ready copy in single experiment and

with single code revision

Parse output[¶](#module-lab.parser "Permalink to this headline")

----------------------------------------------------------------

A parser can be any program that analyzes files in the run’s directory

(e.g. `run.log`) and manipulates the `properties` file in the same

directory.

To make parsing easier, however, you can use the `Parser` class. Here is

an example parser for the FF planner:

../examples/ff/ff-parser.py[¶](#id1 "Permalink to this code")

```

#! /usr/bin/env python

"""

FF example output:

[...]

ff: found legal plan as follows

step 0: UP F0 F1

 1: BOARD F1 P0

 2: DOWN F1 F0

 3: DEPART F0 P0

time spent: 0.00 seconds instantiating 4 easy, 0 hard action templates

 0.00 seconds reachability analysis, yielding 4 facts and 4 actions

 0.00 seconds creating final representation with 4 relevant facts

 0.00 seconds building connectivity graph

 0.00 seconds searching, evaluating 5 states, to a max depth of 2

 0.00 seconds total time

"""

import re

from lab.parser import Parser

def error(content, props):

    if props["planner\_exit\_code"] == 0:

        props["error"] = "plan-found"

    else:

        props["error"] = "unsolvable-or-error"

def coverage(content, props):

    props["coverage"] = int(props["planner\_exit\_code"] == 0)

def get\_plan(content, props):

    # All patterns are parsed before functions are called.

    if props.get("evaluations") is not None:

        props["plan"] = re.findall(r"^(?:step)?\s\*\d+: (.+)$", content, re.M)

def get\_times(content, props):

    props["times"] = re.findall(r"(\d+\.\d+) seconds", content)

def trivially\_unsolvable(content, props):

    props["trivially\_unsolvable"] = int(

        "ff: goal can be simplified to FALSE. No plan will solve it" in content

    )

parser = Parser()

parser.add\_pattern("node", r"node: (.+)\n", type=str, file="driver.log", required=True)

parser.add\_pattern(

    "planner\_exit\_code", r"run-planner exit code: (.+)\n", type=int, file="driver.log"

)

parser.add\_pattern("evaluations", r"evaluating (\d+) states")

parser.add\_function(error)

parser.add\_function(coverage)

parser.add\_function(get\_plan)

parser.add\_function(get\_times)

parser.add\_function(trivially\_unsolvable)

parser.parse()

```

You can add this parser to all runs by using [`add\_parser()`](index.html#lab.experiment.Experiment.add_parser "lab.experiment.Experiment.add_parser"):

```

>>> from pathlib import Path

>>> from lab.experiment import Experiment

>>> exp = Experiment()

>>> # The path can be absolute or relative to the working directory at build time.

>>> parser = Path(\_\_file\_\_).resolve().parents[1] / "examples/ff/ff-parser.py"

>>> exp.add\_parser(parser)

```

All added parsers will be run in the order in which they were added after

executing the run’s commands.

If you need to change your parsers and execute them again, use the

[`add\_parse\_again\_step()`](index.html#lab.experiment.Experiment.add_parse_again_step "lab.experiment.Experiment.add_parse_again_step") method to re-parse

your results.

Run other planners[¶](#run-other-planners "Permalink to this headline")

-----------------------------------------------------------------------

The script below shows how to run the [FF planner](http://fai.cs.uni-saarland.de/hoffmann/ff.html) on a number of

classical planning benchmarks. You can see the available steps with

```

./ff.py

```

Select steps by name or index:

```

./ff.py build

./ff.py 2

./ff.py 3 4

```

You can use this file as a basis for your own experiments. For Fast

Downward experiments, we recommend taking a look at the

<downward.tutorial>.

../examples/ff/ff.py[¶](#id1 "Permalink to this code")

```

#! /usr/bin/env python

"""

Example experiment for the FF planner

(http://fai.cs.uni-saarland.de/hoffmann/ff.html).

"""

import os

import platform

from downward import suites

from downward.reports.absolute import AbsoluteReport

from lab.environments import BaselSlurmEnvironment, LocalEnvironment

from lab.experiment import Experiment

from lab.reports import Attribute, geometric\_mean

# Create custom report class with suitable info and error attributes.

class BaseReport(AbsoluteReport):

    INFO\_ATTRIBUTES = ["time\_limit", "memory\_limit"]

    ERROR\_ATTRIBUTES = [

        "domain",

        "problem",

        "algorithm",

        "unexplained\_errors",

        "error",

        "node",

    ]

NODE = platform.node()

REMOTE = NODE.endswith(".scicore.unibas.ch") or NODE.endswith(".cluster.bc2.ch")

BENCHMARKS\_DIR = os.environ["DOWNWARD\_BENCHMARKS"]

if REMOTE:

    ENV = BaselSlurmEnvironment(email="my.name@unibas.ch")

else:

    ENV = LocalEnvironment(processes=2)

SUITE = ["grid", "gripper:prob01.pddl", "miconic:s1-0.pddl", "mystery:prob07.pddl"]

ATTRIBUTES = [

    "error",

    "plan",

    "times",

    Attribute("coverage", absolute=True, min\_wins=False, scale="linear"),

    Attribute("evaluations", function=geometric\_mean),

    Attribute("trivially\_unsolvable", min\_wins=False),

]

TIME\_LIMIT = 1800

MEMORY\_LIMIT = 2048

# Create a new experiment.

exp = Experiment(environment=ENV)

# Add custom parser for FF.

exp.add\_parser("ff-parser.py")

for task in suites.build\_suite(BENCHMARKS\_DIR, SUITE):

    run = exp.add\_run()

    # Create symbolic links and aliases. This is optional. We

    # could also use absolute paths in add\_command().

    run.add\_resource("domain", task.domain\_file, symlink=True)

    run.add\_resource("problem", task.problem\_file, symlink=True)

    # 'ff' binary has to be on the PATH.

    # We could also use exp.add\_resource().

    run.add\_command(

        "run-planner",

        ["ff", "-o", "{domain}", "-f", "{problem}"],

        time\_limit=TIME\_LIMIT,

        memory\_limit=MEMORY\_LIMIT,

    )

    # AbsoluteReport needs the following properties:

    # 'domain', 'problem', 'algorithm', 'coverage'.

    run.set\_property("domain", task.domain)

    run.set\_property("problem", task.problem)

    run.set\_property("algorithm", "ff")

    # BaseReport needs the following properties:

    # 'time\_limit', 'memory\_limit'.

    run.set\_property("time\_limit", TIME\_LIMIT)

    run.set\_property("memory\_limit", MEMORY\_LIMIT)

    # Every run has to have a unique id in the form of a list.

    # The algorithm name is only really needed when there are

    # multiple algorithms.

    run.set\_property("id", ["ff", task.domain, task.problem])

# Add step that writes experiment files to disk.

exp.add\_step("build", exp.build)

# Add step that executes all runs.

exp.add\_step("start", exp.start\_runs)

# Add step that collects properties from run directories and

# writes them to \*-eval/properties.

exp.add\_fetcher(name="fetch")

# Make a report.

exp.add\_report(BaseReport(attributes=ATTRIBUTES), outfile="report.html")

# Parse the commandline and run the specified steps.

exp.run\_steps()

```

Here is a simple parser for FF:

../examples/ff/ff-parser.py[¶](#id2 "Permalink to this code")

```

#! /usr/bin/env python

"""

FF example output:

[...]

ff: found legal plan as follows

step 0: UP F0 F1

 1: BOARD F1 P0

 2: DOWN F1 F0

 3: DEPART F0 P0

time spent: 0.00 seconds instantiating 4 easy, 0 hard action templates

 0.00 seconds reachability analysis, yielding 4 facts and 4 actions

 0.00 seconds creating final representation with 4 relevant facts

 0.00 seconds building connectivity graph

 0.00 seconds searching, evaluating 5 states, to a max depth of 2

 0.00 seconds total time

"""

import re

from lab.parser import Parser

def error(content, props):

    if props["planner\_exit\_code"] == 0:

        props["error"] = "plan-found"

    else:

        props["error"] = "unsolvable-or-error"

def coverage(content, props):

    props["coverage"] = int(props["planner\_exit\_code"] == 0)

def get\_plan(content, props):

    # All patterns are parsed before functions are called.

    if props.get("evaluations") is not None:

        props["plan"] = re.findall(r"^(?:step)?\s\*\d+: (.+)$", content, re.M)

def get\_times(content, props):

    props["times"] = re.findall(r"(\d+\.\d+) seconds", content)

def trivially\_unsolvable(content, props):

    props["trivially\_unsolvable"] = int(

        "ff: goal can be simplified to FALSE. No plan will solve it" in content

    )

parser = Parser()

parser.add\_pattern("node", r"node: (.+)\n", type=str, file="driver.log", required=True)

parser.add\_pattern(

    "planner\_exit\_code", r"run-planner exit code: (.+)\n", type=int, file="driver.log"

)

parser.add\_pattern("evaluations", r"evaluating (\d+) states")

parser.add\_function(error)

parser.add\_function(coverage)

parser.add\_function(get\_plan)

parser.add\_function(get\_times)

parser.add\_function(trivially\_unsolvable)

parser.parse()

```

Run Singularity images[¶](#run-singularity-images "Permalink to this headline")

-------------------------------------------------------------------------------

The script below shows how to run Singularity planner images using

Downward Lab.

../examples/singularity/singularity-exp.py[¶](#id1 "Permalink to this code")

```

#! /usr/bin/env python

"""

Example experiment for running Singularity/Apptainer planner images.

The time and memory limits set with Lab can be circumvented by solvers that fork

child processes. Their resource usage is not checked. If you're running solvers

that don't check their resource usage like Fast Downward, we recommend using

cgroups or the "runsolver" tool to enforce resource limits. Since setting time

limits for solvers with cgroups is difficult, the experiment below uses the

``runsolver`` tool, which has been used in multiple SAT competitions to enforce

resource limits. For the experiment to run, the runsolver binary needs to be on

the PATH. You can obtain a runsolver copy from

https://github.com/jendrikseipp/runsolver.

Since Singularity (and Apptainer) reserve 1-2 GiB of \*virtual\* memory when

starting the container, we recommend either enforcing a higher virtual memory

limit with ``runsolver`` or limiting RSS memory with ``runsolver`` (like below).

For limiting RSS memory, you can also use `runlim

<https://github.com/arminbiere/runlim>`\_, which is more actively maintained than

runsolver.

A note on running Singularity on clusters: reading large Singularity files over

the network is not optimal, so we recommend copying the images to a local

filesystem (e.g., /tmp/) before running experiments.

"""

import os

from pathlib import Path

import platform

import sys

from downward import suites

from downward.reports.absolute import AbsoluteReport

from lab.environments import BaselSlurmEnvironment, LocalEnvironment

from lab.experiment import Experiment

# Create custom report class with suitable info and error attributes.

class BaseReport(AbsoluteReport):

    INFO\_ATTRIBUTES = []

    ERROR\_ATTRIBUTES = [

        "domain",

        "problem",

        "algorithm",

        "unexplained\_errors",

        "error",

        "node",

    ]

NODE = platform.node()

RUNNING\_ON\_CLUSTER = NODE.endswith((".scicore.unibas.ch", ".cluster.bc2.ch"))

DIR = Path(\_\_file\_\_).resolve().parent

REPO = DIR.parent

IMAGES\_DIR = Path(os.environ["SINGULARITY\_IMAGES"])

assert IMAGES\_DIR.is\_dir(), IMAGES\_DIR

BENCHMARKS\_DIR = os.environ["DOWNWARD\_BENCHMARKS"]

MEMORY\_LIMIT = 3584  # MiB

if RUNNING\_ON\_CLUSTER:

    SUITE = ["depot", "freecell", "gripper", "zenotravel"]

    ENVIRONMENT = BaselSlurmEnvironment(

        partition="infai\_1",

        email="my.name@unibas.ch",

        memory\_per\_cpu="3872M",

        export=["PATH"],

        setup=BaselSlurmEnvironment.DEFAULT\_SETUP,

        # Until recently, we had to load the Singularity module here

        # by adding "module load Singularity/2.6.1 2> /dev/null".

    )

    TIME\_LIMIT = 1800

else:

    SUITE = ["depot:p01.pddl", "gripper:prob01.pddl", "mystery:prob07.pddl"]

    ENVIRONMENT = LocalEnvironment(processes=2)

    TIME\_LIMIT = 5

ATTRIBUTES = [

    "cost",

    "coverage",

    "error",

    "g\_values\_over\_time",

    "run\_dir",

    "raw\_memory",

    "runtime",

    "virtual\_memory",

]

exp = Experiment(environment=ENVIRONMENT)

exp.add\_step("build", exp.build)

exp.add\_step("start", exp.start\_runs)

exp.add\_fetcher(name="fetch")

exp.add\_parser(DIR / "singularity-parser.py")

def get\_image(name):

    planner = name.replace("-", "\_")

    image = IMAGES\_DIR / (name + ".img")

    assert image.is\_file(), image

    return planner, image

IMAGES = [get\_image("fd1906-lama-first")]

for planner, image in IMAGES:

    exp.add\_resource(planner, image, symlink=True)

exp.add\_resource("run\_singularity", DIR / "run-singularity.sh")

exp.add\_resource("filter\_stderr", DIR / "filter-stderr.py")

for planner, \_ in IMAGES:

    for task in suites.build\_suite(BENCHMARKS\_DIR, SUITE):

        run = exp.add\_run()

        run.add\_resource("domain", task.domain\_file, "domain.pddl")

        run.add\_resource("problem", task.problem\_file, "problem.pddl")

        # Use runsolver to limit time and memory. It must be on the system

        # PATH. Important: we cannot use time\_limit and memory\_limit of

        # Lab's add\_command() because setting the same memory limit with

        # runsolver again using setrlimit fails.

        run.add\_command(

            "run-planner",

            [

                "runsolver",

                "--cpu-limit",

                TIME\_LIMIT,

                "--rss-swap-limit",

                MEMORY\_LIMIT,

                "--watcher-data",

                "watch.log",

                "--var",

                "values.log",

                "{run\_singularity}",

                f"{{{planner}}}",

                "{domain}",

                "{problem}",

                "sas\_plan",

            ],

        )

        # Remove temporary files from old Fast Downward versions.

        run.add\_command("rm-tmp-files", ["rm", "-f", "output.sas", "output"])

        run.add\_command("filter-stderr", [sys.executable, "{filter\_stderr}"])

        run.set\_property("domain", task.domain)

        run.set\_property("problem", task.problem)

        run.set\_property("algorithm", planner)

        run.set\_property("id", [planner, task.domain, task.problem])

report = Path(exp.eval\_dir) / f"{exp.name}.html"

exp.add\_report(BaseReport(attributes=ATTRIBUTES), outfile=report)

exp.run\_steps()

```

The experiment script needs a parser and a helper script:

../examples/singularity/singularity-parser.py[¶](#id2 "Permalink to this code")

```

#! /usr/bin/env python

import re

import sys

from lab.parser import Parser

def coverage(content, props):

    props["coverage"] = int("cost" in props)

def unsolvable(content, props):

    # Note that this naive test may easily generate false positives.

    props["unsolvable"] = int(

        not props["coverage"]

        and "Completely explored state space -- no solution!" in content

    )

def parse\_g\_value\_over\_time(content, props):

 """Example line: "[g=6, 16 evaluated, 15 expanded, t=0.00328561s, 22300 KB]" """

    matches = re.findall(

        r"\[g=(\d+), \d+ evaluated, \d+ expanded, t=(.+)s, \d+ KB\]\n", content

    )

    props["g\_values\_over\_time"] = [(float(t), int(g)) for g, t in matches]

def set\_outcome(content, props):

    lines = content.splitlines()

    solved = props["coverage"]

    unsolvable = props["unsolvable"]

    out\_of\_time = int("TIMEOUT=true" in lines)

    out\_of\_memory = int("MEMOUT=true" in lines)

    # runsolver decides "out of time" based on CPU rather than (cumulated)

    # WCTIME.

    if (

        not solved

        and not unsolvable

        and not out\_of\_time

        and not out\_of\_memory

        and props["runtime"] > props["time\_limit"]

    ):

        out\_of\_time = 1

    # In cases where CPU time is very slightly above the threshold so that

    # runsolver didn't kill the planner yet and the planner solved a task

    # just within the limit, runsolver will still record an "out of time".

    # We remove this record. This case also applies to iterative planners.

    # If such planners solve the task, we don't treat them as running out

    # of time.

    if (solved or unsolvable) and (out\_of\_time or out\_of\_memory):

        print("task solved however runsolver recorded an out\_of\_\*")

        print(props)

        out\_of\_time = 0

        out\_of\_memory = 0

    if not solved and not unsolvable:

        props["runtime"] = None

    if solved ^ unsolvable ^ out\_of\_time ^ out\_of\_memory:

        if solved:

            props["error"] = "solved"

        elif unsolvable:

            props["error"] = "unsolvable"

        elif out\_of\_time:

            props["error"] = "out\_of\_time"

        elif out\_of\_memory:

            props["error"] = "out\_of\_memory"

    else:

        print(f"unexpected error: {props}", file=sys.stderr)

        props["error"] = "unexpected-error"

def main():

    print("Running singularity parser")

    parser = Parser()

    parser.add\_pattern(

        "planner\_exit\_code",

        r"run-planner exit code: (.+)\n",

        type=int,

        file="driver.log",

        required=True,

    )

    parser.add\_pattern(

        "node", r"node: (.+)\n", type=str, file="driver.log", required=True

    )

    parser.add\_pattern(

        "planner\_wall\_clock\_time",

        r"run-planner wall-clock time: (.+)s",

        type=float,

        file="driver.log",

        required=True,

    )

    parser.add\_pattern("runtime", r"Singularity runtime: (.+?)s", type=float)

    parser.add\_pattern(

        "time\_limit",

        r"Enforcing CPUTime limit \(soft limit, will send "

        r"SIGTERM then SIGKILL\): (\d+) seconds",

        type=int,

        file="watch.log",

        required=True,

    )

    # Cumulative runtime and virtual memory of the solver and all child processes.

    parser.add\_pattern(

        "runtime", r"WCTIME=(.+)", type=float, file="values.log", required=True

    )

    parser.add\_pattern(

        "virtual\_memory", r"MAXVM=(\d+)", type=int, file="values.log", required=True

    )

    parser.add\_pattern("raw\_memory", r"Peak memory: (\d+) KB", type=int)

    parser.add\_pattern("cost", r"\nFinal value: (.+)\n", type=int)

    parser.add\_function(coverage)

    parser.add\_function(unsolvable)

    parser.add\_function(parse\_g\_value\_over\_time)

    parser.add\_function(set\_outcome, file="values.log")

    parser.parse()

if \_\_name\_\_ == "\_\_main\_\_":

    main()

```

../examples/singularity/run-singularity.sh[¶](#id3 "Permalink to this code")

```

#!/bin/bash

set -euo pipefail

if [[ $# != 4 ]]; then

    echo "usage: $(basename "$0") image domain_file problem_file plan_file" 1>&2

    exit 2

fi

if [ -f $PWD/$4 ]; then

    echo "Error: remove $PWD/$4" 1>&2

    exit 2

fi

# Ensure that strings like "CPU time limit exceeded" and "Killed" are in English.

export LANG=C

set +e

singularity run -C -H "$PWD" "$1" "$PWD/$2" "$PWD/$3" "$4"

set -e

printf "\nRun VAL\n\n"

if [ -f $PWD/$4 ]; then

    echo "Found plan file."

    validate -v "$PWD/$2" "$PWD/$3" "$PWD/$4"

    exit 0

else

    echo "No plan file."

    validate -v "$PWD/$2" "$PWD/$3"

    exit 99

fi

```

[`lab.experiment`](#module-lab.experiment "lab.experiment") — Create experiments[¶](#module-lab.experiment "Permalink to this headline")

----------------------------------------------------------------------------------------------------------------------------------------

### [`Experiment`](#lab.experiment.Experiment "lab.experiment.Experiment")[¶](#experiment "Permalink to this headline")

*class* `lab.experiment.``Experiment`(*path=None*, *environment=None*)[[source]](_modules/lab/experiment.html#Experiment)[¶](#lab.experiment.Experiment "Permalink to this definition")

Base class for Lab experiments.

See [Concepts](index.html#concepts) for a description of how Lab experiments are

structured.

The experiment will be built at *path*. It defaults to

`<scriptdir>/data/<scriptname>/`. E.g., for the script

`experiments/myexp.py`, the default *path* will be

`experiments/data/myexp/`.

*environment* must be an [Environment](#environments)

instance. You can use

[`LocalEnvironment`](#lab.environments.LocalEnvironment "lab.environments.LocalEnvironment") to run your

experiment on a single computer (default). If you have access to

the computer grid in Basel you can use the predefined grid

environment [`BaselSlurmEnvironment`](#lab.environments.BaselSlurmEnvironment "lab.environments.BaselSlurmEnvironment").

Alternatively, you can derive your own class from

[Environment](#environments).

`add_command`(*name*, *command*, *time\_limit=None*, *memory\_limit=None*, *soft\_stdout\_limit=1024*, *hard\_stdout\_limit=10240*, *soft\_stderr\_limit=64*, *hard\_stderr\_limit=10240*, *\*\*kwargs*)[¶](#lab.experiment.Experiment.add_command "Permalink to this definition")

Call an executable.

If invoked on a *run*, this method adds the command to the

**specific** run. If invoked on the experiment, the command is

appended to the list of commands of **all** runs.

*name* is a string describing the command. It must start with a

letter and consist exclusively of letters, numbers, underscores

and hyphens.

*command* has to be a list of strings where the first item is

the executable.

After *time\_limit* seconds the signal SIGXCPU is sent to the

command. The process can catch this signal and exit gracefully.

If it doesn’t catch the SIGXCPU signal, the command is aborted

with SIGKILL after five additional seconds. The time spent by a

command is the sum of time spent across all threads of the

process.

The command is aborted with SIGKILL when it uses more than

*memory\_limit* MiB.

You can limit the log size (in KiB) with a soft and hard limit

for both stdout and stderr. When the soft limit is hit, an

unexplained error is registered for this run, but the command is

allowed to continue running. When the hard limit is hit, the

command is killed with SIGTERM. This signal can be caught and

handled by the process.

By default, there are limits for the log and error output, but

time and memory are not restricted.

All *kwargs* (except `stdin`) are passed to [subprocess.Popen](http://docs.python.org/library/subprocess.html). Instead of

file handles you can also pass filenames for the `stdout` and

`stderr` keyword arguments. Specifying the `stdin` kwarg is

not supported.

```

>>> exp = Experiment()

>>> run = exp.add\_run()

>>> # Add commands to a \*specific\* run.

>>> run.add\_command("solver", ["mysolver", "input-file"], time\_limit=60)

>>> # Add a command to \*all\* runs.

>>> exp.add\_command("cleanup", ["rm", "my-temp-file"])

```

Make sure to call all Python programs from the currently active

Python interpreter, i.e., `sys.executable`. Otherwise, the

system Python version might be used instead of the Python version

from the virtual environment.

```

>>> run.add\_command("myplanner", [sys.executable, "planner.py", "input-file"])

```

`add_fetcher`(*src=None*, *dest=None*, *merge=None*, *name=None*, *filter=None*, *\*\*kwargs*)[[source]](_modules/lab/experiment.html#Experiment.add_fetcher)[¶](#lab.experiment.Experiment.add_fetcher "Permalink to this definition")

Add a step that fetches results from experiment or evaluation

directories into a new or existing evaluation directory.

You can use this method to combine results from multiple

experiments.

*src* can be an experiment or evaluation directory. It defaults

to `exp.path`.

*dest* must be a new or existing evaluation directory. It

defaults to `exp.eval\_dir`. If *dest* already contains

data and *merge* is set to None, the user will be prompted

whether to override the existing data or to merge the old and

new data. Setting *merge* to True or to False has the effect

that the old data is merged or replaced (and the user will not

be prompted).

If no *name* is given, call this step “fetch-`basename(src)`”.

You can fetch only a subset of runs (e.g., runs for specific

domains or algorithms) by passing [`filters`](index.html#lab.reports.Report "lab.reports.Report")

with the *filter* argument.

Example setup:

```

>>> exp = Experiment("/tmp/exp")

```

Fetch all results and write a single combined properties file

to the default evaluation directory (this step is added by

default):

```

>>> exp.add\_fetcher(name="fetch")

```

Merge the results from “other-exp” into this experiment’s

results:

```

>>> exp.add\_fetcher(src="/path/to/other-exp-eval")

```

Fetch only the runs for certain algorithms:

```

>>> exp.add\_fetcher(filter\_algorithm=["algo\_1", "algo\_5"])

```

`add_new_file`(*name*, *dest*, *content*, *permissions=420*)[¶](#lab.experiment.Experiment.add_new_file "Permalink to this definition")

Write *content* to /path/to/exp-or-run/*dest* and make the new file

available to the commands as *name*.

*name* is an alias for the resource in commands. It must start with a

letter and consist exclusively of letters, numbers and underscores.

```

>>> exp = Experiment()

>>> run = exp.add\_run()

>>> run.add\_new\_file("learn", "learn.txt", "a = 5; b = 2; c = 5")

>>> run.add\_command("print-trainingset", ["cat", "{learn}"])

```

`add_parse_again_step`()[[source]](_modules/lab/experiment.html#Experiment.add_parse_again_step)[¶](#lab.experiment.Experiment.add_parse_again_step "Permalink to this definition")

Add a step that copies the parsers from their originally specified

locations to the experiment directory and runs all of them again. This

step overwrites the existing properties file in each run dir.

Do not forget to run the default fetch step again to overwrite

existing data in the -eval dir of the experiment.

`add_parser`(*path\_to\_parser*)[[source]](_modules/lab/experiment.html#Experiment.add_parser)[¶](#lab.experiment.Experiment.add_parser "Permalink to this definition")

Add a parser to each run of the experiment.

Add the parser as a resource to the experiment and add a command

that executes the parser to each run. Since commands are

executed in the order they are added, parsers should be added

after all other commands. If you need to change your parsers and

execute them again you can use the [`add\_parse\_again\_step()`](#lab.experiment.Experiment.add_parse_again_step "lab.experiment.Experiment.add_parse_again_step")

method.

*path\_to\_parser* must be the path to a Python script. The script

is executed in the run directory and manipulates the run’s

“properties” file. The last part of the filename (without the

extension) is used as a resource name. Therefore, it must be

unique among all parsers and other resources. Also, it must

start with a letter and contain only letters, numbers,

underscores and dashes (which are converted to underscores

automatically).

For information about how to write parsers see [Parser](#parsing).

`add_report`(*report*, *name=''*, *eval\_dir=''*, *outfile=''*)[[source]](_modules/lab/experiment.html#Experiment.add_report)[¶](#lab.experiment.Experiment.add_report "Permalink to this definition")

Add *report* to the list of experiment steps.

This method is a shortcut for `add\_step(name, report,

eval\_dir, outfile)` and uses sensible defaults for omitted

arguments.

If no *name* is given, use *outfile* or the *report*’s class name.

By default, use the experiment’s standard *eval\_dir*.

If *outfile* is omitted, compose a filename from *name* and the

*report*’s format. If *outfile* is a relative path, put it under

*eval\_dir*.

```

>>> from downward.reports.absolute import AbsoluteReport

>>> exp = Experiment("/tmp/exp")

>>> exp.add\_report(AbsoluteReport(attributes=["coverage"]))

```

`add_resource`(*name*, *source*, *dest=''*, *symlink=False*)[¶](#lab.experiment.Experiment.add_resource "Permalink to this definition")

Include the file or directory *source* in the experiment or run.

*name* is an alias for the resource in commands. It must start with a

letter and consist exclusively of letters, numbers and underscores.

If you don’t need an alias for the resource, set name=’’.

*source* is copied to /path/to/exp-or-run/*dest*. If *dest* is

omitted, the last part of the path to *source* will be taken as the

destination filename. If you only want an alias for your resource, but

don’t want to copy or link it, set *dest* to None.

Example:

```

>>> exp = Experiment()

>>> exp.add\_resource("planner", "path/to/my-planner")

```

includes my-planner in the experiment directory. You can use

`{planner}` to reference my-planner in a run’s commands:

```

>>> run = exp.add\_run()

>>> run.add\_resource("domain", "path-to/gripper/domain.pddl")

>>> run.add\_resource("task", "path-to/gripper/prob01.pddl")

>>> run.add\_command("plan", ["{planner}", "{domain}", "{task}"])

```

`add_run`(*run=None*)[[source]](_modules/lab/experiment.html#Experiment.add_run)[¶](#lab.experiment.Experiment.add_run "Permalink to this definition")

Schedule *run* to be part of the experiment.

If *run* is None, create a new run, add it to the experiment

and return it.

`add_step`(*name*, *function*, *\*args*, *\*\*kwargs*)[[source]](_modules/lab/experiment.html#Experiment.add_step)[¶](#lab.experiment.Experiment.add_step "Permalink to this definition")

Add a step to the list of experiment steps.

Use this method to add experiment steps like writing the

experiment file to disk, removing directories and publishing

results. To add fetch and report steps, use the convenience

methods [`add\_fetcher()`](#lab.experiment.Experiment.add_fetcher "lab.experiment.Experiment.add_fetcher") and [`add\_report()`](#lab.experiment.Experiment.add_report "lab.experiment.Experiment.add_report").

*name* is a descriptive name for the step. When selecting steps

on the command line, you may either use step names or their

indices.

*function* must be a callable Python object, e.g., a function

or a class implementing \_\_call\_\_.

*args* and *kwargs* will be passed to *function* when the step

is executed.

```

>>> import shutil

>>> import subprocess

>>> from lab.experiment import Experiment

>>> exp = Experiment("/tmp/myexp")

>>> exp.add\_step("build", exp.build)

>>> exp.add\_step("start", exp.start\_runs)

>>> exp.add\_step("rm-eval-dir", shutil.rmtree, exp.eval\_dir)

>>> exp.add\_step("greet", subprocess.call, ["echo", "Hello"])

```

`build`(*write\_to\_disk=True*)[[source]](_modules/lab/experiment.html#Experiment.build)[¶](#lab.experiment.Experiment.build "Permalink to this definition")

Finalize the internal data structures, then write all files

needed for the experiment to disk.

If *write\_to\_disk* is False, only compute the internal data

structures. This is only needed on grids for

FastDownwardExperiments.build() which turns the added algorithms

and benchmarks into Runs.

`eval_dir`[¶](#lab.experiment.Experiment.eval_dir "Permalink to this definition")

Return the name of the default evaluation directory.

This is the directory where the fetched and parsed results will land by

default.

`name`[¶](#lab.experiment.Experiment.name "Permalink to this definition")

Return the directory name of the experiment’s `path`.

`run_steps`()[[source]](_modules/lab/experiment.html#Experiment.run_steps)[¶](#lab.experiment.Experiment.run_steps "Permalink to this definition")

Parse the commandline and run selected steps.

`set_property`(*name*, *value*)[¶](#lab.experiment.Experiment.set_property "Permalink to this definition")

Add a key-value property.

These can be used later, for example, in reports.

```

>>> exp = Experiment()

>>> exp.set\_property("suite", ["gripper", "grid"])

>>> run = exp.add\_run()

>>> run.set\_property("domain", "gripper")

>>> run.set\_property("problem", "prob01.pddl")

```

Each run must have the property *id* which must be a *unique*

list of strings. They determine where the results for this run

will land in the combined properties file.

```

>>> run.set\_property("id", ["algo1", "task1"])

>>> run.set\_property("id", ["algo2", "domain1", "problem1"])

```

`start_runs`()[[source]](_modules/lab/experiment.html#Experiment.start_runs)[¶](#lab.experiment.Experiment.start_runs "Permalink to this definition")

Execute all runs that were added to the experiment.

Depending on the selected environment this method will start

the runs locally or on a computer grid.

#### Custom command line arguments[¶](#custom-command-line-arguments "Permalink to this headline")

`lab.experiment.``ARGPARSER`[¶](#lab.experiment.ARGPARSER "Permalink to this definition")

[ArgumentParser](http://docs.python.org/library/argparse.html)

instance that can be used to add custom command line arguments.

You can import it, add your arguments and call its `parse\_args()`

method to retrieve the argument values. To avoid confusion with step

names you shouldn’t use positional arguments.

Note

Custom command line arguments are only passed to locally

executed steps.

```

from lab.experiment import ARGPARSER

ARGPARSER.add\_argument(

    "--test",

    choices=["yes", "no"],

    required=True,

    dest="test\_run",

    help="run experiment on small suite locally")

args = ARGPARSER.parse\_args()

if args.test\_run:

    print "perform test run"

else:

    print "run real experiment"

```

### [`Run`](#lab.experiment.Run "lab.experiment.Run")[¶](#run "Permalink to this headline")

*class* `lab.experiment.``Run`(*experiment*)[[source]](_modules/lab/experiment.html#Run)[¶](#lab.experiment.Run "Permalink to this definition")

An experiment consists of multiple runs. There should be one run

for each (algorithm, benchmark) pair.

A run consists of one or more commands.

*experiment* must be an [`Experiment`](#lab.experiment.Experiment "lab.experiment.Experiment") instance.

`add_command`(*name*, *command*, *time\_limit=None*, *memory\_limit=None*, *soft\_stdout\_limit=1024*, *hard\_stdout\_limit=10240*, *soft\_stderr\_limit=64*, *hard\_stderr\_limit=10240*, *\*\*kwargs*)[¶](#lab.experiment.Run.add_command "Permalink to this definition")

Call an executable.

If invoked on a *run*, this method adds the command to the

**specific** run. If invoked on the experiment, the command is

appended to the list of commands of **all** runs.

*name* is a string describing the command. It must start with a

letter and consist exclusively of letters, numbers, underscores

and hyphens.

*command* has to be a list of strings where the first item is

the executable.

After *time\_limit* seconds the signal SIGXCPU is sent to the

command. The process can catch this signal and exit gracefully.

If it doesn’t catch the SIGXCPU signal, the command is aborted

with SIGKILL after five additional seconds. The time spent by a

command is the sum of time spent across all threads of the

process.

The command is aborted with SIGKILL when it uses more than

*memory\_limit* MiB.

You can limit the log size (in KiB) with a soft and hard limit

for both stdout and stderr. When the soft limit is hit, an

unexplained error is registered for this run, but the command is

allowed to continue running. When the hard limit is hit, the

command is killed with SIGTERM. This signal can be caught and

handled by the process.

By default, there are limits for the log and error output, but

time and memory are not restricted.

All *kwargs* (except `stdin`) are passed to [subprocess.Popen](http://docs.python.org/library/subprocess.html). Instead of

file handles you can also pass filenames for the `stdout` and

`stderr` keyword arguments. Specifying the `stdin` kwarg is

not supported.

```

>>> exp = Experiment()

>>> run = exp.add\_run()

>>> # Add commands to a \*specific\* run.

>>> run.add\_command("solver", ["mysolver", "input-file"], time\_limit=60)

>>> # Add a command to \*all\* runs.

>>> exp.add\_command("cleanup", ["rm", "my-temp-file"])

```

Make sure to call all Python programs from the currently active

Python interpreter, i.e., `sys.executable`. Otherwise, the

system Python version might be used instead of the Python version

from the virtual environment.

```

>>> run.add\_command("myplanner", [sys.executable, "planner.py", "input-file"])

```

`add_new_file`(*name*, *dest*, *content*, *permissions=420*)[¶](#lab.experiment.Run.add_new_file "Permalink to this definition")

Write *content* to /path/to/exp-or-run/*dest* and make the new file

available to the commands as *name*.

*name* is an alias for the resource in commands. It must start with a

letter and consist exclusively of letters, numbers and underscores.

```

>>> exp = Experiment()

>>> run = exp.add\_run()

>>> run.add\_new\_file("learn", "learn.txt", "a = 5; b = 2; c = 5")

>>> run.add\_command("print-trainingset", ["cat", "{learn}"])

```

`add_resource`(*name*, *source*, *dest=''*, *symlink=False*)[¶](#lab.experiment.Run.add_resource "Permalink to this definition")

Include the file or directory *source* in the experiment or run.

*name* is an alias for the resource in commands. It must start with a

letter and consist exclusively of letters, numbers and underscores.

If you don’t need an alias for the resource, set name=’’.

*source* is copied to /path/to/exp-or-run/*dest*. If *dest* is

omitted, the last part of the path to *source* will be taken as the

destination filename. If you only want an alias for your resource, but

don’t want to copy or link it, set *dest* to None.

Example:

```

>>> exp = Experiment()

>>> exp.add\_resource("planner", "path/to/my-planner")

```

includes my-planner in the experiment directory. You can use

`{planner}` to reference my-planner in a run’s commands:

```

>>> run = exp.add\_run()

>>> run.add\_resource("domain", "path-to/gripper/domain.pddl")

>>> run.add\_resource("task", "path-to/gripper/prob01.pddl")

>>> run.add\_command("plan", ["{planner}", "{domain}", "{task}"])

```

`set_property`(*name*, *value*)[¶](#lab.experiment.Run.set_property "Permalink to this definition")

Add a key-value property.

These can be used later, for example, in reports.

```

>>> exp = Experiment()

>>> exp.set\_property("suite", ["gripper", "grid"])

>>> run = exp.add\_run()

>>> run.set\_property("domain", "gripper")

>>> run.set\_property("problem", "prob01.pddl")

```

Each run must have the property *id* which must be a *unique*

list of strings. They determine where the results for this run

will land in the combined properties file.

```

>>> run.set\_property("id", ["algo1", "task1"])

>>> run.set\_property("id", ["algo2", "domain1", "problem1"])

```

### `CachedRevision`[¶](#cachedrevision "Permalink to this headline")

*class* `lab.cached_revision.``CachedRevision`(*revision\_cache*, *repo*, *rev*, *build\_cmd*, *exclude=None*, *subdir=''*)[[source]](_modules/lab/cached_revision.html#CachedRevision)[¶](#lab.cached_revision.CachedRevision "Permalink to this definition")

Cache compiled revisions of a solver for quick reuse.

* *revision\_cache*: path to revision cache directory.

* *repo*: path to solver repository.

* *rev*: solver revision.

* *build\_cmd*: list with build script and any build options

(e.g., `["./build.py", "release"]`, `["make"]`). Will be executed under

*subdir*.

* *exclude*: list of relative paths under *subdir* that are not needed for

building and running the solver. Instead of this parameter, you can also

use a `.gitattributes` file for Git repositories.

* *subdir*: relative path from *repo* to solver subdir.

The following example caches a Fast Downward revision. When you

use the [`FastDownwardExperiment`](index.html#downward.experiment.FastDownwardExperiment "downward.experiment.FastDownwardExperiment") class, you don’t

need to cache revisions yourself since the class will do it

transparently for you.

```

>>> import os

>>> repo = os.environ["DOWNWARD\_REPO"]

>>> revision\_cache = os.environ.get("DOWNWARD\_REVISION\_CACHE")

>>> if revision\_cache:

...     rev = "main"

...     cr = CachedRevision(

...         revision\_cache, repo, rev, ["./build.py"], exclude=["experiments"]

...     )

...     # cr.cache() # Uncomment to actually cache the code.

...

```

You can now copy the cached repo to your experiment:

```

...     from lab.experiment import Experiment

...     exp = Experiment()

...     dest\_path = os.path.join(exp.path, f"code-{cr.name}")

...     exp.add\_resource(f"solver\_{cr.name}", cr.path, dest\_path)

```

`cache`()[[source]](_modules/lab/cached_revision.html#CachedRevision.cache)[¶](#lab.cached_revision.CachedRevision.cache "Permalink to this definition")

Check out the solver revision to *self.path* and compile the solver.

`get_relative_exp_path`(*relpath=''*)[[source]](_modules/lab/cached_revision.html#CachedRevision.get_relative_exp_path)[¶](#lab.cached_revision.CachedRevision.get_relative_exp_path "Permalink to this definition")

Return a path relative to the experiment directory.

Use this function to find out where files from the cache will be put in

the experiment directory.

### `Parser`[¶](#parser "Permalink to this headline")

*class* `lab.parser.``Parser`[[source]](_modules/lab/parser.html#Parser)[¶](#lab.parser.Parser "Permalink to this definition")

Parse files in the current directory and write results into the

run’s `properties` file.

`add_function`(*function*, *file='run.log'*)[[source]](_modules/lab/parser.html#Parser.add_function)[¶](#lab.parser.Parser.add_function "Permalink to this definition")

Call `function(open(file).read(), properties)` during parsing.

Functions are applied **after** all patterns have been

evaluated and in the order in which they are added to the parser.

The function is passed the file contents and the properties

dictionary. It must manipulate the passed properties

dictionary. The return value is ignored.

Example:

```

>>> import re

>>> from lab.parser import Parser

>>> def parse\_states\_over\_time(content, props):

...     matches = re.findall(r"(.+)s: (\d+) states\n", content)

...     props["states\_over\_time"] = [(float(t), int(s)) for t, s in matches]

...

>>> parser = Parser()

>>> parser.add\_function(parse\_states\_over\_time)

```

You can use `props.add\_unexplained\_error("message")` when your

parsing function detects that something went wrong during the

run.

`add_pattern`(*attribute*, *regex*, *file='run.log'*, *type=<class 'int'>*, *flags=''*, *required=False*)[[source]](_modules/lab/parser.html#Parser.add_pattern)[¶](#lab.parser.Parser.add_pattern "Permalink to this definition")

Look for *regex* in *file*, cast what is found in brackets to

*type* and store it in the properties dictionary under

*attribute*. During parsing roughly the following code will be

executed:

```

contents = open(file).read()

match = re.compile(regex).search(contents)

properties[attribute] = type(match.group(1))

```

*flags* must be a string of Python regular expression flags (see

<https://docs.python.org/3/library/re.html>). E.g., `flags="M"`

lets “^” and “$” match at the beginning and end of each line,

respectively.

If *required* is True and the pattern is not found in *file*,

an error message is printed to stderr.

```

>>> parser = Parser()

>>> parser.add\_pattern("facts", r"Facts: (\d+)", type=int)

```

`parse`()[[source]](_modules/lab/parser.html#Parser.parse)[¶](#lab.parser.Parser.parse "Permalink to this definition")

Search all patterns and apply all functions.

The found values are written to the run’s `properties` file.

### `Environment`[¶](#environment "Permalink to this headline")

*class* `lab.environments.``Environment`(*randomize\_task\_order=True*)[[source]](_modules/lab/environments.html#Environment)[¶](#lab.environments.Environment "Permalink to this definition")

Abstract base class for all environments.

If *randomize\_task\_order* is True (default), tasks for runs are

started in a random order. This is useful to avoid systematic

noise due to, e.g., one of the algorithms being run on a

machine with heavy load. Note that due to the randomization,

run directories may be pristine while the experiment is running

even though the logs say the runs are finished.

*class* `lab.environments.``LocalEnvironment`(*processes=None*, *\*\*kwargs*)[[source]](_modules/lab/environments.html#LocalEnvironment)[¶](#lab.environments.LocalEnvironment "Permalink to this definition")

Environment for running experiments locally on a single machine.

If given, *processes* must be between 1 and #CPUs. If omitted,

it will be set to #CPUs.

See [`Environment`](#lab.environments.Environment "lab.environments.Environment") for inherited

parameters.

*class* `lab.environments.``SlurmEnvironment`(*email=None*, *extra\_options=None*, *partition=None*, *qos=None*, *time\_limit\_per\_task=None*, *memory\_per\_cpu=None*, *cpus\_per\_task=1*, *export=None*, *setup=None*, *\*\*kwargs*)[[source]](_modules/lab/environments.html#SlurmEnvironment)[¶](#lab.environments.SlurmEnvironment "Permalink to this definition")

Abstract base class for Slurm environments.

If the main experiment step is part of the selected steps, the

selected steps are submitted to Slurm. Otherwise, the selected steps

are run locally.

Note

If the steps are run by Slurm, this class writes job files to

the directory `<exppath>-grid-steps` and makes them depend on

one another. Please inspect the \*.log and \*.err files in

this directory if something goes wrong. Since the job files call

the experiment script during execution, it mustn’t be changed

during the experiment.

If *email* is provided and the steps run on the grid, a message will

be sent when the last experiment step finishes.

Use *extra\_options* to pass additional options. The *extra\_options*

string may contain newlines. The first example below uses only a given

set of nodes (additional nodes will be used if the given ones don’t

satisfy the resource constraints). The second example shows show to

specify a project account (needed on NSC if you’re part of multiple

projects).

```

extra\_options="#SBATCH --nodelist=ase[1-5,7,10]"

extra\_options="#SBATCH --account=snic2021-5-330"

```

*partition* must be a valid Slurm partition name. In Basel you

can choose from

* “infai\_1”: 24 nodes with 16 cores, 64GB memory, 500GB Sata (default)

* “infai\_2”: 24 nodes with 20 cores, 128GB memory, 240GB SSD

*qos* must be a valid Slurm QOS name. In Basel this must be

“normal”.

*time\_limit\_per\_task* sets the wall-clock time limit for each Slurm task.

The BaselSlurmEnvironment subclass uses a default of “0”, i.e., no limit.

(Note that there may still be an external limit set in slurm.conf.)

The TetralithEnvironment class uses a default of “24:00:00”, i.e., 24

hours. This is because in certain situations, the scheduler prefers to

schedule tasks shorter than 24 hours.

*memory\_per\_cpu* must be a string specifying the memory

allocated for each core. The string must end with one of the

letters K, M or G. The default is “3872M”. The value for

*memory\_per\_cpu* should not surpass the amount of memory that is

available per core, which is “3872M” for infai\_1 and “6354M” for

infai\_2. Processes that surpass the *memory\_per\_cpu* limit are

terminated with SIGKILL. To impose a soft limit that can be

caught from within your programs, you can use the

`memory\_limit` kwarg of

[`add\_command()`](#lab.experiment.Run.add_command "lab.experiment.Run.add_command"). Fast Downward users

should set memory limits via the `driver\_options`.

Slurm limits the memory with cgroups. Unfortunately, this often

fails on our nodes, so we set our own soft memory limit for all

Slurm jobs. We derive the soft memory limit by multiplying the

value denoted by the *memory\_per\_cpu* parameter with 0.98 (the

Slurm config file contains “AllowedRAMSpace=99” and we add some

slack). We use a soft instead of a hard limit so that child

processes can raise the limit.

*cpus\_per\_task* sets the number of cores to be allocated per Slurm

task (default: 1).

Examples that reserve the maximum amount of memory available per core:

```

>>> env1 = BaselSlurmEnvironment(partition="infai\_1", memory\_per\_cpu="3872M")

>>> env2 = BaselSlurmEnvironment(partition="infai\_2", memory\_per\_cpu="6354M")

```

Example that reserves 12 GiB of memory on infai\_1:

```

>>> # 12 \* 1024 / 3872 = 3.17 -> round to next int -> 4 cores per task

>>> # 12G / 4 = 3G per core

>>> env = BaselSlurmEnvironment(

...     partition="infai\_1",

...     memory\_per\_cpu="3G",

...     cpus\_per\_task=4,

... )

```

Example that reserves 12 GiB of memory on infai\_2:

```

>>> # 12 \* 1024 / 6354 = 1.93 -> round to next int -> 2 cores per task

>>> # 12G / 2 = 6G per core

>>> env = BaselSlurmEnvironment(

...     partition="infai\_2",

...     memory\_per\_cpu="6G",

...     cpus\_per\_task=2,

... )

```

Use *export* to specify a list of environment variables that

should be exported from the login node to the compute nodes

(default: [“PATH”]).

You can alter the environment in which the experiment runs with

the *setup* argument. If given, it must be a string of Bash

commands. Example:

```

# Load Singularity module.

setup="module load Singularity/2.6.1 2> /dev/null"

```

Slurm limits the number of job array tasks. You must set the

appropriate value for your cluster in the *MAX\_TASKS* class

variable. Lab groups ceil(runs/MAX\_TASKS) runs in one array

task.

See [`Environment`](#lab.environments.Environment "lab.environments.Environment") for inherited

parameters.

*class* `lab.environments.``BaselSlurmEnvironment`(*email=None*, *extra\_options=None*, *partition=None*, *qos=None*, *time\_limit\_per\_task=None*, *memory\_per\_cpu=None*, *cpus\_per\_task=1*, *export=None*, *setup=None*, *\*\*kwargs*)[[source]](_modules/lab/environments.html#BaselSlurmEnvironment)[¶](#lab.environments.BaselSlurmEnvironment "Permalink to this definition")

Environment for Basel’s AI group.

*class* `lab.environments.``TetralithEnvironment`(*email=None*, *extra\_options=None*, *partition=None*, *qos=None*, *time\_limit\_per\_task=None*, *memory\_per\_cpu=None*, *cpus\_per\_task=1*, *export=None*, *setup=None*, *\*\*kwargs*)[[source]](_modules/lab/environments.html#TetralithEnvironment)[¶](#lab.environments.TetralithEnvironment "Permalink to this definition")

Environment for the NSC Tetralith cluster in Linköping.

### Various[¶](#various "Permalink to this headline")

`lab.``__version__`[¶](#lab.__version__ "Permalink to this definition")

Lab version number. A “+” is appended to all non-tagged revisions.

`lab.reports` – Make reports[¶](#lab-reports-make-reports "Permalink to this headline")

---------------------------------------------------------------------------------------

`lab.reports.``arithmetic_mean`(*values*)[[source]](_modules/lab/reports.html#arithmetic_mean)[¶](#lab.reports.arithmetic_mean "Permalink to this definition")

Compute the arithmetic mean of a sequence of numbers.

```

>>> arithmetic\_mean([20, 30, 70])

40.0

```

`lab.reports.``geometric_mean`(*values*)[[source]](_modules/lab/reports.html#geometric_mean)[¶](#lab.reports.geometric_mean "Permalink to this definition")

Compute the geometric mean of a sequence of numbers.

```

>>> round(geometric\_mean([2, 8]), 2)

4.0

```

*class* `lab.reports.``Attribute`(*name*, *absolute=False*, *min\_wins=True*, *function=None*, *functions=None*, *scale=None*, *digits=2*)[[source]](_modules/lab/reports.html#Attribute)[¶](#lab.reports.Attribute "Permalink to this definition")

A string subclass for attributes in reports.

Use this class if your **custom** attribute needs a non-default

value for:

* *absolute*: if False, only include tasks for which all task

runs have values in a per-domain table (e.g. `coverage` is

absolute, whereas `expansions` is not, because we can’t

compare algorithms A and B for task X if B has no value for

`expansions`).

* *min\_wins*: set to True if a smaller value for this attribute

is better, to False if a higher value is better and to None

if values can’t be compared. (E.g., *min\_wins* is False for

`coverage`, but it is True for `expansions`).

* *function*: the function used to aggregate

values of multiple runs for this attribute, for example, in

domain reports. Defaults to `sum()`.

* *functions*: deprecated. Pass a single *function* instead.

* *scale*: default scaling. Can be one of “linear”, “log” and

“symlog”. If *scale* is None (default), the reports will

choose the scaling.

* *digits*: number of digits after the decimal point.

The `downward` package automatically uses appropriate

settings for most attributes.

```

>>> avg\_h = Attribute("avg\_h", min\_wins=False)

>>> abstraction\_done = Attribute(

...     "abstraction\_done", absolute=True, min\_wins=False

... )

```

*class* `lab.reports.``Report`(*attributes=None*, *format='html'*, *filter=None*, *\*\*kwargs*)[[source]](_modules/lab/reports.html#Report)[¶](#lab.reports.Report "Permalink to this definition")

Base class for all reports.

Inherit from this or a child class to implement a custom report.

Depending on the type of output you want to make, you will have

to overwrite the [`write()`](#lab.reports.Report.write "lab.reports.Report.write"), [`get\_text()`](#lab.reports.Report.get_text "lab.reports.Report.get_text") or

[`get\_markup()`](#lab.reports.Report.get_markup "lab.reports.Report.get_markup") method.

*attributes* is the list of attributes you want to include in

your report. If omitted, use all numerical attributes. Globbing

characters \* and ? are allowed. Example:

```

>>> report = Report(attributes=["coverage", "translator\_\*"])

```

When a report is made, both the available and the selected

attributes are printed on the commandline.

*format* can be one of e.g. html, tex, wiki (MediaWiki), doku

(DokuWiki), pmw (PmWiki), moin (MoinMoin) and txt (Plain text).

Subclasses may allow additional formats.

If given, *filter* must be a function or a list of functions

that are passed a dictionary of a run’s attribute keys and

values. Filters must return True, False or a new dictionary.

Depending on the returned value, the run is included or excluded

from the report, or replaced by the new dictionary,

respectively.

Filters for properties can be given in shorter form without

defining a function. To include only runs where attribute

`foo` has value v, use `filter\_foo=v`. To include only runs

where attribute `foo` has value v1, v2 or v3, use

`filter\_foo=[v1, v2, v3]`.

Filters are applied sequentially, i.e., the first filter is

applied to all runs before the second filter is executed.

Filters given as `filter\_\*` kwargs are applied *after* all

filters passed via the `filter` kwarg.

Examples:

Include only the “cost” attribute in a LaTeX report:

```

>>> report = Report(attributes=["cost"], format="tex")

```

Only include successful runs in the report:

```

>>> report = Report(filter\_coverage=1)

```

Only include runs in the report where the initial h value is

at most 100:

```

>>> def low\_init\_h(run):

...     return run["initial\_h\_value"] <= 100

...

>>> report = Report(filter=low\_init\_h)

```

Only include runs from “blocks” and “barman” with a timeout:

```

>>> report = Report(filter\_domain=["blocks", "barman"], filter\_search\_timeout=1)

```

Add a new attribute:

```

>>> def add\_expansions\_per\_time(run):

...     expansions = run.get("expansions")

...     time = run.get("search\_time")

...     if expansions is not None and time:

...         run["expansions\_per\_time"] = expansions / time

...     return run

...

>>> report = Report(

...     attributes=["expansions\_per\_time"], filter=[add\_expansions\_per\_time]

... )

```

Rename, filter and sort algorithms:

```

>>> def rename\_algorithms(run):

...     name = run["algorithm"]

...     paper\_names = {"lama11": "LAMA 2011", "fdss\_sat1": "FDSS 1"}

...     run["algorithm"] = paper\_names[name]

...     return run

...

```

```

>>> # We want LAMA 2011 to be the leftmost column.

>>> # filter\_\* filters are evaluated last, so we use the updated

>>> # algorithm names here.

>>> algorithms = ["LAMA 2011", "FDSS 1"]

>>> report = Report(filter=rename\_algorithms, filter\_algorithm=algorithms)

```

`__call__`(*eval\_dir*, *outfile*)[[source]](_modules/lab/reports.html#Report.__call__)[¶](#lab.reports.Report.__call__ "Permalink to this definition")

Make the report.

This method is called automatically when the report step is

executed. It loads the data and calls [`write()`](#lab.reports.Report.write "lab.reports.Report.write").

*eval\_dir* must be a path to an evaluation directory containing

a `properties` file.

The report will be written to *outfile*.

`get_markup`()[[source]](_modules/lab/reports.html#Report.get_markup)[¶](#lab.reports.Report.get_markup "Permalink to this definition")

Return [txt2tags](http://txt2tags.org/) markup for the report.

`get_text`()[[source]](_modules/lab/reports.html#Report.get_text)[¶](#lab.reports.Report.get_text "Permalink to this definition")

Return text (e.g., HTML, LaTeX, etc.) for the report.

By default this method calls [`get\_markup()`](#lab.reports.Report.get_markup "lab.reports.Report.get_markup") and converts

the markup to the desired output *format*.

`write`()[[source]](_modules/lab/reports.html#Report.write)[¶](#lab.reports.Report.write "Permalink to this definition")

Write the report files.

By default this method calls [`get\_text()`](#lab.reports.Report.get_text "lab.reports.Report.get_text") and writes the

obtained text to *outfile*.

Overwrite this method if you want to write the report file(s)

directly. You should write them to *self.outfile*.

*class* `lab.reports.filter.``FilterReport`(*\*\*kwargs*)[[source]](_modules/lab/reports/filter.html#FilterReport)[¶](#lab.reports.filter.FilterReport "Permalink to this definition")

Filter properties files.

This report only applies the given filter and writes a new

properties file to the given output destination.

```

>>> def remove\_openstacks(run):

...     return "openstacks" not in run["domain"]

...

```

```

>>> from lab.experiment import Experiment

>>> report = FilterReport(filter=remove\_openstacks)

>>> exp = Experiment()

>>> exp.add\_report(report, outfile="path/to/new/properties")

```

`downward.experiment` — Fast Downward experiment[¶](#downward-experiment-fast-downward-experiment "Permalink to this headline")

-------------------------------------------------------------------------------------------------------------------------------

Note

The [`FastDownwardExperiment`](#downward.experiment.FastDownwardExperiment "downward.experiment.FastDownwardExperiment") class makes it easy to write

“standard” experiments with little boilerplate code, but it assumes a rigid

experiment structure: it only allows you to run each added algorithm on each

added task, and individual runs cannot easily be customized. An example for

this is the [2020-09-11-A-cg-vs-ff.py](https://github.com/aibasel/lab/tree/main/examples/downward/2020-09-11-A-cg-vs-ff.py)

experiment. If you need more flexibility, you can use the

[`lab.experiment.Experiment`](index.html#lab.experiment.Experiment "lab.experiment.Experiment") class instead and fill it by using

[`FastDownwardAlgorithm`](#downward.experiment.FastDownwardAlgorithm "downward.experiment.FastDownwardAlgorithm"),

[`FastDownwardRun`](#downward.experiment.FastDownwardRun "downward.experiment.FastDownwardRun"),

[`CachedFastDownwardRevision`](#downward.cached_revision.CachedFastDownwardRevision "downward.cached_revision.CachedFastDownwardRevision"), and [`Task`](#downward.suites.Task "downward.suites.Task") objects. The [2020-09-11-B-bounded-cost.py](https://github.com/aibasel/lab/tree/main/examples/downward/2020-09-11-B-bounded-cost.py)

script shows an example. All of these classes are documented below.

*class* `downward.experiment.``FastDownwardExperiment`(*path=None*, *environment=None*, *revision\_cache=None*)[[source]](_modules/downward/experiment.html#FastDownwardExperiment)[¶](#downward.experiment.FastDownwardExperiment "Permalink to this definition")

Conduct a Fast Downward experiment.

The most important methods for customizing an experiment are

[`add\_algorithm()`](#downward.experiment.FastDownwardExperiment.add_algorithm "downward.experiment.FastDownwardExperiment.add_algorithm"), [`add\_suite()`](#downward.experiment.FastDownwardExperiment.add_suite "downward.experiment.FastDownwardExperiment.add_suite"), [`add\_parser()`](index.html#lab.experiment.Experiment.add_parser "lab.experiment.Experiment.add_parser"),

[`add\_step()`](index.html#lab.experiment.Experiment.add_step "lab.experiment.Experiment.add_step") and [`add\_report()`](index.html#lab.experiment.Experiment.add_report "lab.experiment.Experiment.add_report").

Note

To build the experiment, execute its runs and fetch the results,

add the following steps:

```

>>> exp = FastDownwardExperiment()

>>> exp.add\_step("build", exp.build)

>>> exp.add\_step("start", exp.start\_runs)

>>> exp.add\_fetcher(name="fetch")

```

See [`lab.experiment.Experiment`](index.html#lab.experiment.Experiment "lab.experiment.Experiment") for an explanation of

the *path* and *environment* parameters.

*revision\_cache* is the directory for caching Fast Downward

revisions. It defaults to `<scriptdir>/data/revision-cache`.

This directory can become very large since each revision uses

about 30 MB.

```

>>> from lab.environments import BaselSlurmEnvironment

>>> env = BaselSlurmEnvironment(email="my.name@unibas.ch")

>>> exp = FastDownwardExperiment(environment=env)

```

You can add parsers with [`add\_parser()`](index.html#lab.experiment.Experiment.add_parser "lab.experiment.Experiment.add_parser"). See

[Parser](index.html#parsing) for how to write custom parsers and

[Bundled parsers](#downward-parsers) for the list of built-in parsers. Which

parsers you should use depends on the algorithms you’re running.

For single-search experiments, we recommend adding the following

parsers in this order:

```

>>> exp.add\_parser(exp.EXITCODE\_PARSER)

>>> exp.add\_parser(exp.TRANSLATOR\_PARSER)

>>> exp.add\_parser(exp.SINGLE\_SEARCH\_PARSER)

>>> exp.add\_parser(exp.PLANNER\_PARSER)

```

`add_algorithm`(*name*, *repo*, *rev*, *component\_options*, *build\_options=None*, *driver\_options=None*)[[source]](_modules/downward/experiment.html#FastDownwardExperiment.add_algorithm)[¶](#downward.experiment.FastDownwardExperiment.add_algorithm "Permalink to this definition")

Add a Fast Downward algorithm to the experiment, i.e., a

planner configuration in a given repository at a given

revision.

*name* is a string describing the algorithm (e.g.

`"issue123-lmcut"`).

*repo* must be a path to a Fast Downward repository.

*rev* must be a valid revision in the given repository (e.g.,

`"e9c2370e6"`, `"my-branch"`, `"issue123"`).

*component\_options* must be a list of strings. By default these

options are passed to the search component. Use

`"--translate-options"`, `"--preprocess-options"` or

`"--search-options"` within the component options to override

the default for the following options, until overridden again.

If given, *build\_options* must be a list of strings. They will be

passed to the `build.py` script. Options can be build names

(e.g., `"releasenolp"`), `build.py` options (e.g.,

`"--debug"`) or options for Make. If *build\_options* is omitted,

the `"release"` version is built.

If given, *driver\_options* must be a list of strings. They will be

passed to the `fast-downward.py` script. See `fast-downward.py

--help` for available options. The list is always prepended with

`["--validate", "--overall-time-limit", "30m",

"--overall-memory-limit', "3584M"]`. Specifying custom limits

overrides the default limits.

Example experiment setup:

```

>>> import os

>>> exp = FastDownwardExperiment()

>>> repo = os.environ["DOWNWARD\_REPO"]

>>> rev = "main"

```

Run iPDB using the latest revision on the main branch:

```

>>> exp.add\_algorithm("ipdb", repo, rev, ["--search", "astar(ipdb())"])

```

Run blind search in debug mode:

```

>>> exp.add\_algorithm(

...     "blind",

...     repo,

...     rev,

...     ["--search", "astar(blind())"],

...     build\_options=["--debug"],

...     driver\_options=["--debug"],

... )

```

Run LAMA-2011 with custom planner time limit:

```

>>> exp.add\_algorithm(

...     "lama",

...     repo,

...     rev,

...     [],

...     driver\_options=[

...         "--alias",

...         "seq-saq-lama-2011",

...         "--overall-time-limit",

...         "5m",

...     ],

... )

```

`add_suite`(*benchmarks\_dir*, *suite*)[[source]](_modules/downward/experiment.html#FastDownwardExperiment.add_suite)[¶](#downward.experiment.FastDownwardExperiment.add_suite "Permalink to this definition")

Add PDDL or SAS+ benchmarks to the experiment.

*benchmarks\_dir* must be a path to a benchmark directory. It must

contain domain directories, which in turn hold PDDL or SAS+ files

(ending with “.pddl” or “.sas”).

*suite* must be a list of domain or domain:task names.

```

>>> benchmarks\_dir = os.environ["DOWNWARD\_BENCHMARKS"]

>>> exp = FastDownwardExperiment()

>>> exp.add\_suite(benchmarks\_dir, ["depot", "gripper"])

>>> exp.add\_suite(benchmarks\_dir, ["gripper:prob01.pddl"])

>>> exp.add\_suite(benchmarks\_dir, ["rubiks-cube:p01.sas"])

```

One source for benchmarks is

<https://github.com/aibasel/downward-benchmarks>. After cloning the repo,

you can generate suites with the `suites.py` script. We recommend

using the suite `optimal\_strips` for optimal STRIPS planners and

`satisficing` for satisficing planners:

```

# Create standard optimal planning suite. $

path/to/downward-benchmarks/suites.py optimal\_strips ['airport',

..., 'zenotravel']

```

Then you can copy the generated list into your experiment script:

```

>>> exp.add\_suite(benchmarks\_dir, ["airport", "zenotravel"])

```

### Bundled parsers[¶](#bundled-parsers "Permalink to this headline")

The following constants are paths to default parsers that can be passed

to [`exp.add\_parser()`](index.html#lab.experiment.Experiment.add_parser "lab.experiment.Experiment.add_parser"). The

“Used attributes” and “Parsed attributes” lists describe the

dependencies between the parsers.

`FastDownwardExperiment.``EXITCODE_PARSER`[¶](#downward.experiment.FastDownwardExperiment.EXITCODE_PARSER "Permalink to this definition")

Parsed attributes: “error”, “planner\_exit\_code”, “unsolvable”.

`FastDownwardExperiment.``TRANSLATOR_PARSER`[¶](#downward.experiment.FastDownwardExperiment.TRANSLATOR_PARSER "Permalink to this definition")

Parsed attributes: “translator\_peak\_memory”, “translator\_time\_done”, etc.

`FastDownwardExperiment.``SINGLE_SEARCH_PARSER`[¶](#downward.experiment.FastDownwardExperiment.SINGLE_SEARCH_PARSER "Permalink to this definition")

Parsed attributes: “coverage”, “memory”, “total\_time”, etc.

`FastDownwardExperiment.``ANYTIME_SEARCH_PARSER`[¶](#downward.experiment.FastDownwardExperiment.ANYTIME_SEARCH_PARSER "Permalink to this definition")

Parsed attributes: “cost”, “cost:all”, “coverage”.

`FastDownwardExperiment.``PLANNER_PARSER`[¶](#downward.experiment.FastDownwardExperiment.PLANNER_PARSER "Permalink to this definition")

Used attributes: “memory”, “total\_time”,

“translator\_peak\_memory”, “translator\_time\_done”.

Parsed attributes: “node”, “planner\_memory”, “planner\_time”,

“planner\_wall\_clock\_time”, “score\_planner\_memory”, “score\_planner\_time”.

*class* `downward.experiment.``FastDownwardAlgorithm`(*name: str*, *cached\_revision: downward.cached\_revision.CachedFastDownwardRevision*, *driver\_options*, *component\_options*)[[source]](_modules/downward/experiment.html#FastDownwardAlgorithm)[¶](#downward.experiment.FastDownwardAlgorithm "Permalink to this definition")

A Fast Downward algorithm is the combination of revision, driver options and

component options.

`cached_revision` *= None*[¶](#downward.experiment.FastDownwardAlgorithm.cached_revision "Permalink to this definition")

An instance of [`CachedFastDownwardRevision`](#downward.cached_revision.CachedFastDownwardRevision "downward.cached_revision.CachedFastDownwardRevision").

`component_options` *= None*[¶](#downward.experiment.FastDownwardAlgorithm.component_options "Permalink to this definition")

Component options, e.g., `["--search", "astar(lmcut())"]`.

`driver_options` *= None*[¶](#downward.experiment.FastDownwardAlgorithm.driver_options "Permalink to this definition")

Driver options, e.g., `["--build", "debug"]`.

`name` *= None*[¶](#downward.experiment.FastDownwardAlgorithm.name "Permalink to this definition")

Algorithm name, e.g., `"rev123:astar-lmcut"`.

*class* `downward.experiment.``FastDownwardRun`(*exp: lab.experiment.Experiment*, *algo: downward.experiment.FastDownwardAlgorithm*, *task: downward.suites.Task*)[[source]](_modules/downward/experiment.html#FastDownwardRun)[¶](#downward.experiment.FastDownwardRun "Permalink to this definition")

An experiment run that uses *algo* to solve *task*.

See [`Run`](index.html#lab.experiment.Run "lab.experiment.Run") for inherited methods.

*class* `downward.cached_revision.``CachedFastDownwardRevision`(*revision\_cache*, *repo*, *rev*, *build\_options*, *subdir=''*)[[source]](_modules/downward/cached_revision.html#CachedFastDownwardRevision)[¶](#downward.cached_revision.CachedFastDownwardRevision "Permalink to this definition")

This class represents Fast Downward checkouts.

It provides methods for caching compiled revisions, so they can be reused

quickly in different experiments.

* *revision\_cache*: Path to revision cache.

* *repo*: Path to Fast Downward repository.

* *rev*: Fast Downward revision.

* *build\_options*: List of build.py options.

* *subdir*: relative path from *repo* to Fast Downward subdir.

`cache`()[¶](#downward.cached_revision.CachedFastDownwardRevision.cache "Permalink to this definition")

Check out the solver revision to *self.path* and compile the solver.

`get_relative_exp_path`(*relpath=''*)[¶](#downward.cached_revision.CachedFastDownwardRevision.get_relative_exp_path "Permalink to this definition")

Return a path relative to the experiment directory.

Use this function to find out where files from the cache will be put in

the experiment directory.

### `downward.suites` — Select benchmarks[¶](#downward-suites-select-benchmarks "Permalink to this headline")

*class* `downward.suites.``Task`(*domain: str*, *problem: str*, *problem\_file*, *domain\_file=None*, *properties=None*)[[source]](_modules/downward/suites.html#Task)[¶](#downward.suites.Task "Permalink to this definition")

*domain* and *problem* are the display names of the domain and

problem, *domain\_file* and *problem\_file* are paths to the

respective files on the disk. If *domain\_file* is not given,

assume that *problem\_file* is a SAS task.

*properties* may be a dictionary of entries that should be

added to the properties file of each run that uses this

problem.

```

>>> task = Task(

...     "gripper",

...     "p01.pddl",

...     problem\_file="/path/to/prob01.pddl",

...     domain\_file="/path/to/domain.pddl",

...     properties={"relaxed": False},

... )

```

`downward.suites.``build_suite`(*benchmarks\_dir*, *descriptions*)[[source]](_modules/downward/suites.html#build_suite)[¶](#downward.suites.build_suite "Permalink to this definition")

Compute a list of [`Task`](#downward.suites.Task "downward.suites.Task") objects.

The path *benchmarks\_dir* must contain a subdir for each domain.

*descriptions* must be a list of domain or problem descriptions:

```

build\_suite(benchmarks\_dir, ["gripper", "grid:prob01.pddl"])

```

`downward.reports` — Fast Downward reports[¶](#downward-reports-fast-downward-reports "Permalink to this headline")

-------------------------------------------------------------------------------------------------------------------

### Tables[¶](#tables "Permalink to this headline")

*class* `downward.reports.``PlanningReport`(*\*\*kwargs*)[[source]](_modules/downward/reports.html#PlanningReport)[¶](#downward.reports.PlanningReport "Permalink to this definition")

Base class for planner reports.

See [`Report`](index.html#lab.reports.Report "lab.reports.Report") for inherited parameters.

You can filter and modify runs for a report with

[`filters`](index.html#lab.reports.Report "lab.reports.Report"). For example, you can include only

a subset of algorithms or compute new attributes. If you provide

a list for *filter\_algorithm*, it will be used to determine the

order of algorithms in the report.

```

>>> # Use a filter function to select algorithms.

>>> def only\_blind\_and\_lmcut(run):

...     return run["algorithm"] in ["blind", "lmcut"]

...

>>> report = PlanningReport(filter=only\_blind\_and\_lmcut)

```

```

>>> # Use "filter\_algorithm" to select and \*order\* algorithms.

>>> report = PlanningReport(filter\_algorithm=["lmcut", "blind"])

```

[`Filters`](index.html#lab.reports.Report "lab.reports.Report") can be very helpful so we

recommend reading up on them to use their full potential.

Subclasses can use the member variable `problem\_runs` to access the

experiment data. It is a dictionary mapping from tasks (i.e.,

`(domain, problem)` pairs) to the runs for that task. Each run is a

dictionary that maps from attribute names to values.

```

>>> class MinRuntimePerTask(PlanningReport):

...     def get\_text(self):

...         map = {}

...         for (domain, problem), runs in self.problem\_runs.items():

...             times = [run.get("planner\_time") for run in runs]

...             times = [t for t in times if t is not None]

...             map[(domain, problem)] = min(times) if times else None

...         return str(map)

...

```

You may want to override the following class attributes in subclasses:

`PREDEFINED_ATTRIBUTES` *= ['cost', 'coverage', 'dead\_ends', 'evaluations', 'expansions', 'generated', 'initial\_h\_value', 'plan\_length', 'planner\_time', 'quality', 'score\_\*', 'search\_time', 'total\_time', 'unsolvable']*[¶](#downward.reports.PlanningReport.PREDEFINED_ATTRIBUTES "Permalink to this definition")

List of predefined `Attribute` instances. For example, if

PlanningReport receives `attributes=['coverage']`, it converts

the plain string `'coverage'` to the attribute instance

`Attribute('coverage', absolute=True, min\_wins=False, scale='linear')`.

`ERROR_ATTRIBUTES` *= ['domain', 'problem', 'algorithm', 'unexplained\_errors', 'error', 'planner\_wall\_clock\_time', 'raw\_memory', 'node']*[¶](#downward.reports.PlanningReport.ERROR_ATTRIBUTES "Permalink to this definition")

Attributes shown in the unexplained-errors table.

`INFO_ATTRIBUTES` *= ['local\_revision', 'global\_revision', 'build\_options', 'driver\_options', 'component\_options']*[¶](#downward.reports.PlanningReport.INFO_ATTRIBUTES "Permalink to this definition")

Attributes shown in the algorithm info table.

*class* `downward.reports.absolute.``AbsoluteReport`(*\*\*kwargs*)[[source]](_modules/downward/reports/absolute.html#AbsoluteReport)[¶](#downward.reports.absolute.AbsoluteReport "Permalink to this definition")

Report absolute values for the selected attributes.

This report should be part of all your Fast Downward experiments as

it includes a table of unexplained errors, e.g. invalid solutions,

segmentation faults, etc.

```

>>> from downward.experiment import FastDownwardExperiment

>>> exp = FastDownwardExperiment()

>>> exp.add\_report(AbsoluteReport(attributes=["expansions"]), outfile="report.html")

```

Example output:

> 

> 

> 

> 

> 

> 

> 

> | expansions | hFF | hCEA |

> | --- | --- | --- |

> | gripper | 118 | 72 |

> | zenotravel | 21 | 17 |

> 

> 

> 

*class* `downward.reports.taskwise.``TaskwiseReport`(*\*\*kwargs*)[[source]](_modules/downward/reports/taskwise.html#TaskwiseReport)[¶](#downward.reports.taskwise.TaskwiseReport "Permalink to this definition")

For each task report all selected attributes in a single row.

If the experiment contains more than one algorithm, use

`filter\_algorithm=["my\_algorithm"]` to select exactly one algorithm

for the report.

```

>>> from downward.experiment import FastDownwardExperiment

>>> exp = FastDownwardExperiment()

>>> exp.add\_report(

...     TaskwiseReport(

...         attributes=["expansions", "search\_time"], filter\_algorithm=["lmcut"]

...     )

... )

```

Example output:

> 

> 

> 

> 

> 

> 

> 

> |  | expansions | search\_time |

> | --- | --- | --- |

> | grid:prob01.pddl | 118234 | 20.02 |

> | gripper:prob01.pddl | 21938 | 17.58 |

> 

> 

> 

*class* `downward.reports.compare.``ComparativeReport`(*algorithm\_pairs*, *\*\*kwargs*)[[source]](_modules/downward/reports/compare.html#ComparativeReport)[¶](#downward.reports.compare.ComparativeReport "Permalink to this definition")

Compare pairs of algorithms.

See [`AbsoluteReport`](#downward.reports.absolute.AbsoluteReport "downward.reports.absolute.AbsoluteReport")

for inherited parameters.

*algorithm\_pairs* is the list of algorithm pairs you want to

compare.

All columns in the report will be arranged such that the

compared algorithms appear next to each other. After the two

columns containing absolute values for the compared algorithms,

a third column (“Diff”) is added showing the difference between

the two values.

Algorithms may appear in multiple comparisons. Algorithms not

mentioned in *algorithm\_pairs* are not included in the report.

If you want to compare algorithms A and B, instead of a pair

`('A', 'B')` you may pass a triple `('A', 'B', 'A vs.

B')`. The third entry of the triple will be used as the name

of the corresponding “Diff” column.

For example, if the properties file contains algorithms A, B, C

and D and *algorithm\_pairs* is `[('A', 'B', 'Diff BA'), ('A',

'C')]` the resulting columns will be A, B, Diff BA (contains B

- A), A, C , Diff (contains C - A).

Example:

```

>>> from downward.experiment import FastDownwardExperiment

>>> exp = FastDownwardExperiment()

>>> algorithm\_pairs = [("default-lmcut", "issue123-lmcut", "Diff lmcut")]

>>> exp.add\_report(ComparativeReport(algorithm\_pairs, attributes=["coverage"]))

```

Example output:

> 

> 

> 

> 

> 

> 

> 

> 

> | coverage | default-lmcut | issue123-lmcut | Diff lmcut |

> | --- | --- | --- | --- |

> | depot | 15 | 17 | 2 |

> | gripper | 7 | 6 | -1 |

> 

> 

> 

### Plots[¶](#plots "Permalink to this headline")

*class* `downward.reports.scatter.``ScatterPlotReport`(*relative=False*, *show\_missing=True*, *get\_category=None*, *title=None*, *scale=None*, *xlabel=''*, *ylabel=''*, *matplotlib\_options=None*, *\*\*kwargs*)[[source]](_modules/downward/reports/scatter.html#ScatterPlotReport)[¶](#downward.reports.scatter.ScatterPlotReport "Permalink to this definition")

Generate a scatter plot for an attribute.

If *relative* is False, create a “standard” scatter plot with a

diagonal line. If *relative* is True, create a relative scatter

plot where each point *(x, y)* corresponds to a task for which

the first algorithm yields a value of *x* and the second

algorithm yields *x \* y*. Relative scatter plots are less common

in the literature, but often show small differences between

algorithms better than “standard” scatter plots.

The keyword argument *attributes* must contain exactly one

attribute.

Use the *filter\_algorithm* keyword argument to select exactly

two algorithms (see example below).

If *show\_missing* is False, we only draw a point for an

algorithm pair if both algorithms have a value.

*get\_category* can be a function that takes **two** runs

(dictionaries of properties) and returns a category name. This

name is used to group the points in the plot. If there is more

than one group, a legend is automatically added. Runs for which

this function returns None are shown in a default category and

are not contained in the legend. For example, to group by

domain:

```

>>> def domain\_as\_category(run1, run2):

...     # run2['domain'] has the same value, because we always

...     # compare two runs of the same problem.

...     return run1["domain"]

...

```

Example grouping by difficulty:

```

>>> def improvement(run1, run2):

...     time1 = run1.get("search\_time", 1800)

...     time2 = run2.get("search\_time", 1800)

...     if time1 > time2:

...         return "better"

...     if time1 == time2:

...         return "equal"

...     return "worse"

...

```

```

>>> from downward.experiment import FastDownwardExperiment

>>> exp = FastDownwardExperiment()

>>> exp.add\_report(

...     ScatterPlotReport(attributes=["search\_time"], get\_category=improvement)

... )

```

Example comparing the number of expanded states for two

algorithms:

```

>>> exp.add\_report(

...     ScatterPlotReport(

...         attributes=["expansions\_until\_last\_jump"],

...         filter\_algorithm=["algorithm-1", "algorithm-2"],

...         get\_category=domain\_as\_category,

...         format="png",  # Use "tex" for pgfplots output.

...     ),

...     name="scatterplot-expansions",

... )

```

The inherited *format* parameter can be set to ‘png’ (default),

‘eps’, ‘pdf’, ‘pgf’ (needs matplotlib 1.2) or ‘tex’. For the

latter a pgfplots plot is created.

If *title* is given it will be used for the name of the plot.

Otherwise, the only given attribute will be the title. If none

is given, there will be no title.

*scale* can have the values ‘linear’, ‘log’ or ‘symlog’. If

omitted, a sensible default will be used for some standard

attributes and ‘log’ otherwise. Relative scatter plots always

use a logarithmic scaling for the *y* axis.

*xlabel* and *ylabel* are the axis labels.

*matplotlib\_options* may be a dictionary of matplotlib rc

parameters (see <http://matplotlib.org/users/customizing.html>):

```

>>> from downward.reports.scatter import ScatterPlotReport

>>> matplotlib\_options = {

...     "font.family": "serif",

...     "font.weight": "normal",

...     # Used if more specific sizes not set.

...     "font.size": 20,

...     "axes.labelsize": 20,

...     "axes.titlesize": 30,

...     "legend.fontsize": 22,

...     "xtick.labelsize": 10,

...     "ytick.labelsize": 10,

...     "lines.markersize": 10,

...     "lines.markeredgewidth": 0.25,

...     "lines.linewidth": 1,

...     # Width and height in inches.

...     "figure.figsize": [8, 8],

...     "savefig.dpi": 100,

... }

>>> report = ScatterPlotReport(

...     attributes=["initial\_h\_value"], matplotlib\_options=matplotlib\_options

... )

```

You can see the full list of matplotlib options and their

defaults by executing

```

import matplotlib

print(matplotlib.rcParamsDefault)

```

[![_images/example-scatter-plot.png](_images/example-scatter-plot.png)](_images/example-scatter-plot.png)

Concepts[¶](#concepts "Permalink to this headline")

---------------------------------------------------

An **experiment** consists of multiple **steps**. Most experiments will

have steps for building and executing the experiment:

```

>>> from lab.experiment import Experiment

>>> exp = Experiment()

>>> exp.add\_step("build", exp.build)

>>> exp.add\_step("start", exp.start\_runs)

```

Moreover, there are usually steps for **fetching** the results and making

**reports**:

```

>>> from lab.reports import Report

>>> exp.add\_fetcher(name="fetch")

>>> exp.add\_report(Report(attributes=["error"]))

```

The “build” step creates all necessary files for running the experiment in

the **experiment directory**. After the “start” step has finished running

the experiment, we can fetch the result from the experiment directory to

the **evaluation directory**. All reports only operate on evaluation

directories.

An experiment usually also has multiple **runs**, one for each pair of

algorithm and benchmark.

When calling [`start\_runs()`](index.html#lab.experiment.Experiment.start_runs "lab.experiment.Experiment.start_runs"), all **runs** part of the

experiment are executed. You can add runs with the [`add\_run()`](index.html#lab.experiment.Experiment.add_run "lab.experiment.Experiment.add_run")

method. Each run needs a unique ID and at least one **command**:

```

>>> for algo in ["algo1", "algo2"]:

...     for value in range(10):

...         run = exp.add\_run()

...         run.set\_property("id", [algo, str(value)])

...         run.add\_command("solve", [algo, str(value)])

```

You can pass the names of selected steps to your experiment script

or use `--all` to execute all steps. At the end of your script,

call `exp.run\_steps()` to parse the commandline and execute the

selected steps.

Changelog[¶](#changelog "Permalink to this headline")

-----------------------------------------------------

### v7.4 (2023-08-18)[¶](#v7-4-2023-08-18 "Permalink to this headline")

#### Lab[¶](#lab "Permalink to this headline")

* Require *revision\_cache* parameter in [`CachedRevision`](index.html#lab.cached_revision.CachedRevision "lab.cached_revision.CachedRevision") constructor (Jendrik Seipp).

* Add *subdir* option for [`CachedRevision`](index.html#lab.cached_revision.CachedRevision "lab.cached_revision.CachedRevision") to support solvers at deeper levels of a repo (Jendrik Seipp).

* Add [`CachedRevision.get\_relative\_exp\_path()`](index.html#lab.cached_revision.CachedRevision.get_relative_exp_path "lab.cached_revision.CachedRevision.get_relative_exp_path") method to query where cache artefacts will land in the experiment directory (Jendrik Seipp).

* Document [`CachedRevision`](index.html#lab.cached_revision.CachedRevision "lab.cached_revision.CachedRevision") class and stabilize its API (Jendrik Seipp).

* Only use documented classes and functions in example experiments (Jendrik Seipp).

#### Downward Lab[¶](#downward-lab "Permalink to this headline")

* Add *subdir* option for [`CachedFastDownwardRevision`](index.html#downward.cached_revision.CachedFastDownwardRevision "downward.cached_revision.CachedFastDownwardRevision") to support Fast Downward checkouts at deeper levels of a repo (Jendrik Seipp).

* Make [`FastDownwardAlgorithm`](index.html#downward.experiment.FastDownwardAlgorithm "downward.experiment.FastDownwardAlgorithm"), [`FastDownwardRun`](index.html#downward.experiment.FastDownwardRun "downward.experiment.FastDownwardRun") and [`CachedFastDownwardRevision`](index.html#downward.cached_revision.CachedFastDownwardRevision "downward.cached_revision.CachedFastDownwardRevision") classes part of the documented, stable API (Jendrik Seipp).

* Describe [two main alternatives](index.html#downward-experiment) for running Fast Downward experiments (Jendrik Seipp).

### v7.3 (2023-03-03)[¶](#v7-3-2023-03-03 "Permalink to this headline")

#### Lab[¶](#id1 "Permalink to this headline")

* Transparently handle xz-compressed properties files (Jendrik Seipp).

* Add CI tests for Python 3.11 (Jendrik Seipp).

#### Downward Lab[¶](#id2 "Permalink to this headline")

* Adapt code for Matplotlib version 3.7 (Jendrik Seipp).

### v7.2 (2022-10-09)[¶](#v7-2-2022-10-09 "Permalink to this headline")

#### Lab[¶](#id3 "Permalink to this headline")

* Raise minimum supported Python version to 3.7 (Jendrik Seipp).

* Add support for Python 3.10 (Jendrik Seipp).

* Apply parsing functions in the order in which they were added (Jendrik Seipp).

* For contributors: document pre-commit hook in `CONTRIBUTING.md` file (Jendrik Seipp).

#### Downward Lab[¶](#id4 "Permalink to this headline")

* Parse peak memory in anytime search parser (Jendrik Seipp).

* Only store “planner\_memory” and “planner\_time” attributes for successful planner

runs (Jendrik Seipp).

* Add fully customizable example planner experiment without `FastDownwardExperiment` class (Jendrik Seipp).

* Show how to group domain directories in example Fast Downward experiment (Jendrik Seipp).

### v7.1 (2022-06-20)[¶](#v7-1-2022-06-20 "Permalink to this headline")

#### Lab[¶](#id5 "Permalink to this headline")

* Revamp Singularity example experiment: use `runsolver` to limit resource usage

(Silvan Sievers and Jendrik Seipp).

#### Downward Lab[¶](#id6 "Permalink to this headline")

* Fix header sizes in HTML reports (Jendrik Seipp).

* Include domains in attribute overview tables even if none of their tasks has an

attribute value for all algorithms (Jendrik Seipp).

* Compute “score\_planner\_time” and “score\_planner\_memory” attributes in planner

parser (Jendrik Seipp).

* Only consider files ending with “.pddl” and “.sas” when building suites (Jendrik Seipp).

* Explicitly left-align non-numeric cells to avoid \multicolumn entries in Latex output

(Jendrik Seipp).

### v7.0 (2021-10-24)[¶](#v7-0-2021-10-24 "Permalink to this headline")

#### Lab[¶](#id7 "Permalink to this headline")

* Remove support for Mercurial repositories (Jendrik Seipp).

#### Downward Lab[¶](#id8 "Permalink to this headline")

* Fix rules for finding domain files for airport and psr-small domains (Silvan Sievers).

* Add more ticks on y axis in relative plots (Jendrik Seipp).

### v6.5 (2021-09-27)[¶](#v6-5-2021-09-27 "Permalink to this headline")

#### Lab[¶](#id9 "Permalink to this headline")

* Allow rerunning experiments. This is useful if some runs were never started,

for example, due to grid node failures. All runs that have already been started

are skipped. For more information see the corresponding [FAQ](index.html#faq)

(Jendrik Seipp).

#### Downward Lab[¶](#id10 "Permalink to this headline")

* Slightly generalize rules for finding domain files, adapted from Fast Downward

(Silvan Sievers).

### v6.4 (2021-07-06)[¶](#v6-4-2021-07-06 "Permalink to this headline")

#### Lab[¶](#id11 "Permalink to this headline")

* Add `TetralithEnvironment` for the NSC cluster in Linköping (Jendrik Seipp).

* Automatically group multiple runs into one Slurm task when the number

of runs exceeds the maximum number of Slurm tasks (Jendrik Seipp).

* Add `time\_limit\_per\_task` parameter to `SlurmEnvironment` (Jendrik Seipp).

* Add `cpus\_per\_task` parameter to `SlurmEnvironment` (#98, Lucas Galery Käser).

* Catch OverflowError when casting large ints to floats (#95, Silvan Sievers).

#### Downward Lab[¶](#id12 "Permalink to this headline")

* None.

### v6.3 (2021-02-14)[¶](#v6-3-2021-02-14 "Permalink to this headline")

#### Lab[¶](#id13 "Permalink to this headline")

* Use long Git revision hashes for revision cache. The short ones differ in length

between Git versions (Jendrik Seipp).

* Run continuous integration tests for Python 3.9 (Jendrik Seipp).

#### Downward Lab[¶](#id14 "Permalink to this headline")

* Remove “revision\_summary” column from info table (Jendrik Seipp).

### v6.2 (2020-10-20)[¶](#v6-2-2020-10-20 "Permalink to this headline")

#### Lab[¶](#id15 "Permalink to this headline")

* Reports: round values to desired precision before determining colors (Jendrik Seipp).

* Restructure and extend documentation (Jendrik Seipp).

* For developers: run CI tests on Ubuntu 20.04 in addition to 18.04 (Jendrik Seipp).

#### Downward Lab[¶](#id16 "Permalink to this headline")

* Allow adding SAS+ files with `FastDownwardExperiment.add\_suite()` (Jendrik Seipp).

### v6.1 (2020-09-15)[¶](#v6-1-2020-09-15 "Permalink to this headline")

#### Lab[¶](#id17 "Permalink to this headline")

* Take float precision into account when highlighting table cells (Jendrik Seipp).

* Allow serializing pathlib.Path objects into JSON files (Jendrik Seipp).

* For developers: add `.github/CONTRIBUTING.md` file (Jendrik Seipp).

* For developers: separate tests for Singularity and FF example experiments from other tests (Jendrik Seipp).

* For developers: skip `cached\_revision` doctests if `DOWNWARD\_REVISION\_CACHE` variable is not set (Jendrik Seipp).

* For developers: use f-strings in code (Jendrik Seipp).

#### Downward Lab[¶](#id18 "Permalink to this headline")

* Print number of tasks above and below separator lines in scatter plots (Jendrik Seipp).

* Ignore tasks for which runs have been filtered out in aggregate reports (Jendrik Seipp).

* Fix order of bracketed task counts per domain in table reports (Jendrik Seipp).

* Gracefully handle empty scatter plots (Jendrik Seipp).

* Make `score\_\*` attributes absolute, i.e., include tasks for which not all algorithms

have a value in aggregations (Jendrik Seipp).

### v6.0 (2020-04-05)[¶](#v6-0-2020-04-05 "Permalink to this headline")

#### Lab[¶](#id19 "Permalink to this headline")

* Bump minimum Python version to 3.6.

* Move `CachedRevision` from `downward` to `lab` package (Thomas Keller).

Please note that the interface to the class is experimental and may change

in the future. Feedback is welcome!

* Let tests fail if any example experiment produces unexplained errors.

#### Downward Lab[¶](#id20 "Permalink to this headline")

* No changes.

### v5.5 (2020-03-13)[¶](#v5-5-2020-03-13 "Permalink to this headline")

#### Lab[¶](#id21 "Permalink to this headline")

* Sort numbers with suffixes (5K, 2M, 8G) and “infinity” correctly in tables.

* Gracefully handle missing “info” or “summary” tables in HTML reports.

* Abort if a function is passed to a `filter\_\*` kwarg.

* Abort if a filter checks missing attribute names

(e.g., when passing `filter\_algorithms` instead of `filter\_algorithm`).

#### Downward Lab[¶](#id22 "Permalink to this headline")

* Add example experiment for running Singularity planner images.

### v5.4 (2020-03-01)[¶](#v5-4-2020-03-01 "Permalink to this headline")

#### Lab[¶](#id23 "Permalink to this headline")

* Use newer txt2tags version and remove bundled copy.

* Call parsers with active Python interpreter.

* Don’t call deprecated `time.clock()` (removed in Python 3.8).

* Don’t add Lab to `PYTHONPATH` in `BaselSlurmEnvironment`.

#### Downward Lab[¶](#id24 "Permalink to this headline")

* Revision cache: only delete “misc” and “experiments” dirs if they exist (Maximilian Fickert).

### v5.3 (2020-02-03)[¶](#v5-3-2020-02-03 "Permalink to this headline")

#### Lab[¶](#id25 "Permalink to this headline")

* Format source code with black (<https://github.com/psf/black>).

* Fix filters: retrieve new run ID from modified runs (Silvan Sievers).

#### Downward Lab[¶](#id26 "Permalink to this headline")

* Remove call to `rm -f output.sas`. Newer Fast Downward versions remove the temporary file

automatically. If you want to keep the file, add `"--keep-sas-file"` to the `driver\_options`.

* Fix ScatterPlotReport: skip None values in max() computation (Silvan Sievers).

* Fix ScatterPlotReport: place diagonal line correctly even if axis scales differ.

### v5.2 (2020-01-07)[¶](#v5-2-2020-01-07 "Permalink to this headline")

#### Lab[¶](#id27 "Permalink to this headline")

* Use line buffering for run.err files.

#### Downward Lab[¶](#id28 "Permalink to this headline")

* Preserve line breaks for error logs in tables.

* If an error log in a table has more than 100 lines, omit surplus lines from the middle of the log.

* Always print the number of runs with unexplained errors when generating any type of report.

### v5.1 (2019-12-10)[¶](#v5-1-2019-12-10 "Permalink to this headline")

#### Lab[¶](#id29 "Permalink to this headline")

* Test Lab on Python 3.8.

* Use active Python version to call run files in local experiments.

#### Downward Lab[¶](#id30 "Permalink to this headline")

* Support Fast Downward Git repos (Patrick Ferber).

### v5.0 (2019-12-04)[¶](#v5-0-2019-12-04 "Permalink to this headline")

#### Lab[¶](#id31 "Permalink to this headline")

* Deprecate support for Python versions 2.7 to 3.5.

* Allow only a single aggregation function for `Attribute` objects.

* If there is only a single HTML table, show it when the page loads.

* Remove broken `--log-level` command line parameter. You can call

`tools.configure\_logging(logging.DEBUG)` to enable debug messages instead.

* Pass old hard memory limit when setting soft memory limit.

#### Downward Lab[¶](#id32 "Permalink to this headline")

* Scatter plots:

	+ Add *relative* parameter for drawing relative scatter plots.

	+ Draw points for algorithm pairs with missing values on axis boundaries.

	+ Allow drawing negative values on linear and symlog axes.

	+ Remove *xscale* and *yscale* parameters in favor of a new *scale* parameter.

	+ Fold `PlotReport` class into `ScatterPlotReport`.

	+ Simplify code by letting Matplotlib compute axis limits automatically.

### v4.2 (2019-09-27)[¶](#v4-2-2019-09-27 "Permalink to this headline")

#### Lab[¶](#id33 "Permalink to this headline")

* Upload to PyPI. Install Lab and Downward Lab with `pip install lab`.

* Add support for running Lab in Python virtual environments (Guillem).

* Parser scripts don’t have to be executable anymore, but they must be Python scripts.

#### Downward Lab[¶](#id34 "Permalink to this headline")

* Abort if two algorithms are identical, i.e., use the same revision, build config and commandline options.

* Scatter plot report: include tasks for which both algorithms have no data if `show\_missing=True`.

### v4.1 (2019-06-03)[¶](#v4-1-2019-06-03 "Permalink to this headline")

* Add support for Python 3. Lab now supports Python 2.7 and Python >= 3.5.

### v4.0 (2019-02-19)[¶](#v4-0-2019-02-19 "Permalink to this headline")

#### Lab[¶](#id35 "Permalink to this headline")

* Parser: don’t try to parse missing files. Print message to stdout instead.

* Add soft memory limit of “memory\_per\_cpu \* 0.98” for Slurm runs to safeguard against cgroup failures.

* Abort if report contains duplicate attribute names.

* Make reports even if fetcher detects unexplained errors.

* Use `flags=''` for [`lab.parser.Parser.add\_pattern()`](index.html#lab.parser.Parser.add_pattern "lab.parser.Parser.add_pattern") by default again.

* Include node names in standard reports and warn if report mixes runs from different partitions.

* Add new example experiment using a simple vertex cover solver.

* `BaselSlurmEnvironment`: don’t load Python 2.7.11 since it might conflict with an already loaded module.

* Raise default `nice` value to 5000.

#### Downward Lab[¶](#id36 "Permalink to this headline")

* Support new Fast Downward exitcodes (Silvan).

* Parse “planner\_wall\_clock\_time” attribute in planner parser.

* Include “planner\_wall\_clock\_time” and “raw\_memory” attributes in unexplained errors table.

* Make PlanningReport more generic by letting derived classes override the new

`PREDEFINED\_ATTRIBUTES`, `INFO\_ATTRIBUTES` and `ERROR\_ATTRIBUTES` class members (Augusto).

* Don’t compute the “quality” attribute automatically. The docs and `showcase-options.py` show

how to add the two filters that together add the IPC quality score to each run.

### v3.0 (2018-07-10)[¶](#v3-0-2018-07-10 "Permalink to this headline")

#### Lab[¶](#id37 "Permalink to this headline")

* Add [`exp.add\_parser()`](index.html#lab.experiment.Experiment.add_parser "lab.experiment.Experiment.add_parser") method. See also [Parser](index.html#parsing) (Silvan).

* Add [`exp.add\_parse\_again\_step()`](index.html#lab.experiment.Experiment.add_parse_again_step "lab.experiment.Experiment.add_parse_again_step") method for running parsers again (Silvan).

* Require that the `build`, `start\_runs` and `fetch` steps are added explicitly (see [`Experiment`](index.html#lab.experiment.Experiment "lab.experiment.Experiment")).

* Remove *required* argument from `add\_resource()`. All resources are now required.

* Use stricter naming rules for commands and resources. See respective `add\_\*` methods for details.

* Use `required=False` and `flags='M'` by default for [`lab.parser.Parser.add\_pattern()`](index.html#lab.parser.Parser.add_pattern "lab.parser.Parser.add_pattern").

* Only support custom command line arguments for locally executed steps.

* Log errors to stderr.

* Log exit codes and wall-clock times of commands to driver.log.

* Add unexplained error if driver.log is empty.

* Let fetcher fetch `properties` and `static-properties` files.

* Remove deprecated possibility of passing Step objects to `add\_step()`.

* Remove deprecated `exp.\_\_call\_\_()` method.

#### Downward Lab[¶](#id38 "Permalink to this headline")

* Add “planner\_timer” and “planner\_memory” attributes.

* Reorganize parsers and don’t add any parser implicitly. See [Bundled parsers](index.html#downward-parsers).

* Add anytime-search parser that parses only “cost”, “cost:all” and “coverage”.

* Revise and simplify single-search parser.

* Parse new Fast Downward exit codes (<http://issues.fast-downward.org/issue739>).

* Don’t exclude (obsolete) “benchmarks” directory when caching revisions.

* Only copy “raw\_memory” value to “memory” when “total\_time” is present.

* Rename “fast-downward” command to “planner”.

* Make “error” attribute optional for reports.

### v2.3 (2018-04-12)[¶](#v2-3-2018-04-12 "Permalink to this headline")

#### Lab[¶](#id39 "Permalink to this headline")

* BaselSlurmEnvironment: Use `infai\_1` and `normal` as default Slurm partition and QOS.

* Remove `OracleGridEngineEnvironment`.

#### Downward Lab[¶](#id40 "Permalink to this headline")

* Use `--overall-time-limit=30m` and `--overall-memory-limit=3584M` for all Fast Downward runs by default.

* Don’t add `-j` option to build options (`build.py` now uses all CPUs automatically).

### v2.2 (2018-03-16)[¶](#v2-2-2018-03-16 "Permalink to this headline")

#### Lab[¶](#id41 "Permalink to this headline")

* Print run and task IDs during local experiments.

* Make warnings and error messages more informative.

* Abort after fetch step if fetcher finds unexplained errors.

* Improve examples and docs.

#### Downward Lab[¶](#id42 "Permalink to this headline")

* Don’t parse preprocessor logs anymore.

* Make regular expressions stricter in parsers.

* Don’t complain if SAS file is missing.

### v2.1 (2017-11-27)[¶](#v2-1-2017-11-27 "Permalink to this headline")

#### Lab[¶](#id43 "Permalink to this headline")

* Add BaselSlurmEnvironment (Florian).

* Support running experiments in virtualenv (Shuwa).

* Redirect output to `driver.log` and `driver.err` as soon as possible.

* Store all observed unexplained errors instead of a single one (Silvan).

* Report unexplained error if `run.err` or `driver.err` contain output.

* Report unexplained error if “error” attribute is missing.

* Add configurable soft and hard limits for output to `run.log` and `run.err`.

* Record grid node for each run and add it to warnings table.

* Omit toprule and bottomrule in LaTeX tables.

* Add `lab.reports.Table.set\_row\_order()` method.

* Only escape text in table cells if it doesn’t contain LaTeX or HTML markup.

* Allow run filters to change a run’s ID (needed for renaming algorithms).

* Add `merge` kwarg to `add\_fetcher()` (Silvan).

* Exit with returncode 1 if fetcher finds unexplained errors.

* Let fetcher show warning if `slurm.err` is not empty.

* Include content of `slurm.err` in reports if it contains text.

* Add continuous integration testing.

* Add `--skip-experiments` option for `tests/run-tests` script.

* Clean up code.

* Polish documentation.

#### Downward Lab[¶](#id44 "Permalink to this headline")

* For each error outcome show number of runs with that outcome in summary table and dedicated tables.

* Add standalone exit code parser. Allow removing translate and search parsers (Silvan).

* Allow passing `Problem` instances to `FastDownwardExperiment.add\_suite()` (Florian).

* Don’t filter duplicate coordinates in scatter plots.

* Don’t round scatter plot coordinates.

* Remove output.sas instead of compressing it.

* Fix scatter plots for multiple categories **and** the default `None` category (Silvan).

### v2.0 (2017-01-09)[¶](#v2-0-2017-01-09 "Permalink to this headline")

#### Lab[¶](#id45 "Permalink to this headline")

* Show warning and ask for action when evaluation dir already exists.

* Add `scale` parameter to Attribute. It is used by the plot reports.

* Add `digits` parameter to Attribute for specifying the number of digits after the decimal point.

* Pass name, function, args and kwargs to `exp.add\_step()`. Deprecate passing Step objects.

* After calling `add\_resource("mynick", ...)`, use resource in commands with “{mynick}”.

* Call: make `name` parameter mandatory, rename `mem\_limit` kwarg to `memory\_limit`.

* Store grid job files in `<exp-dir>-grid-steps`.

* Use common `run-dispatcher` script for local and remote experiments.

* LocalEnvironment: support randomizing task order (enabled by default).

* Make `path` parameter optional for all experiments.

* Warn if steps are listed explicitly and `--all` is used.

* Change main experiment step name from “start” to “run”.

* Deprecate `exp()`. Use `exp.run\_steps()` instead.

* Don’t filter `None` values in `lab.reports` helper functions.

* Make logging clearer.

* Add example FF experiment.

* Remove deprecated code (e.g. predefined Step objects, `tools.sendmail()`).

* Remove `Run.require\_resource()`. All resources have always been available for all runs.

* Fetcher: remove `write\_combined\_props` parameter.

* Remove `Sequence` class.

* Parser: remove `key\_value\_patterns` parameter. A better solution is in the works.

* Remove `tools.overwrite\_dir()` and `tools.get\_command\_output()`.

* Remove `lab.reports.minimum()`, `lab.reports.maximum()`, `lab.reports.stddev()`.

* Move `lab.reports.prod()` to `lab.tools.product()`.

* Rename `lab.reports.gm()` to `lab.reports.geometric\_mean()` and

`lab.reports.avg()` to `lab.reports.arithmetic\_mean()`.

* Many speed improvements and better error messages.

* Rewrite docs.

#### Downward Lab[¶](#id46 "Permalink to this headline")

* Always validate plans. Previous Lab versions don’t add `--validate`

since older Fast Downward versions don’t support it.

* HTML reports: hide tables by default, add buttons for toggling visibility.

* Unify “score\_\*”, “quality” and “coverage” attributes: assign values in range [0, 1]

and compute only sum and no average.

* Don’t print tables on commandline.

* Remove DownwardExperiment and other deprecated code.

* Move `FastDownwardExperiment` into `downward/experiment.py`.

* Rename `config` attribute to `algorithm`. Remove `config\_nick` attribute.

* Change call name from “search” to “fast-downward”.

* Remove “memory\_capped”, and “id\_string” attributes.

* Report raw memory in “unexplained errors” table.

* Parser: remove `group` argument from `add\_pattern()`, and always use group 1.

* Remove `cache\_dir` parameter. Add `revision\_cache` parameter to `FastDownwardExperiment`.

* Fetcher: remove `copy\_all` option.

* Remove predefined benchmark suites.

* Remove IpcReport, ProblemPlotReport, RelativeReport, SuiteReport and TimeoutReport.

* Rename CompareConfigsReport to ComparativeReport.

* Remove possibility to add `\_relative` to an attribute to obtain relative results.

* Apply filters sequentially instead of interleaved.

* PlanningReport: remove `derived\_properties` parameter. Use two filters

instead: one for caching results, the other for adding new properties

(see `QualityFilters` in `downward/reports/\_\_init\_\_.py`).

* PlotReport: use fixed legend location, remove `category\_styles` option.

* AbsoluteReport: remove `colored` parameter and always color HTML reports.

* Don’t use domain links in Latex reports.

* AbsoluteReport: Remove `resolution` parameter and always use `combined` resolution.

* Rewrite docs.

### v1.12 (2017-01-09)[¶](#v1-12-2017-01-09 "Permalink to this headline")

#### Downward Lab[¶](#id47 "Permalink to this headline")

* Only compress “output” file if it exists.

* Preprocess parser: make legacy preprocessor output optional.

### v1.11 (2016-12-15)[¶](#v1-11-2016-12-15 "Permalink to this headline")

#### Lab[¶](#id48 "Permalink to this headline")

* Add bitbucket-pipelines.yml for continuous integration testing.

#### Downward Lab[¶](#id49 "Permalink to this headline")

* Add IPC 2014 benchmark suites (Silvan).

* Set `min\_wins=False` for `dead\_ends` attribute.

* Fit coordinates better into plots.

* Add finite\_sum() function and use it for `initial\_h\_value` (Silvan).

* Update example scripts for repos without benchmarks.

* Update docs.

### v1.10 (2015-12-11)[¶](#v1-10-2015-12-11 "Permalink to this headline")

#### Lab[¶](#id50 "Permalink to this headline")

* Add `permissions` parameter to [`lab.experiment.Experiment.add\_new\_file()`](index.html#lab.experiment.Experiment.add_new_file "lab.experiment.Experiment.add_new_file").

* Add default parser which checks that log files are not bigger than 100 MB. Maybe we’ll make this configurable in the future.

* Ensure that resource names are not shared between runs and experiment.

* Show error message if resource names are not unique.

* Table: don’t format list items. This allows us to keep the quotes for configuration lists.

#### Downward Lab[¶](#id51 "Permalink to this headline")

* Cleanup `downward.suites`: update suite names, add STRIPS and

ADL versions of all IPCs. We recommend selecting a subset of domains

manually to only run your code on “interesting” benchmarks. As a

starting point you can use the suites `suite\_optimal\_strips` or

`suite\_satisficing`.

### v1.9.1 (2015-11-12)[¶](#v1-9-1-2015-11-12 "Permalink to this headline")

#### Downward Lab[¶](#id52 "Permalink to this headline")

* Always prepend build options with `-j<num\_cpus>`.

* Fix: Use correct revisions in `FastDownwardExperiment`.

* Don’t abort parser if resource limits can’t be found (support old planner versions).

### v1.9 (2015-11-07)[¶](#v1-9-2015-11-07 "Permalink to this headline")

#### Lab[¶](#id53 "Permalink to this headline")

* Add [`lab.experiment.Experiment.add\_command()`](index.html#lab.experiment.Experiment.add_command "lab.experiment.Experiment.add_command") method.

* Add [`lab.\_\_version\_\_`](index.html#lab.__version__ "lab.__version__") string.

* Explicitly remove support for Python 2.6.

#### Downward Lab[¶](#id54 "Permalink to this headline")

* Add [`downward.experiment.FastDownwardExperiment`](index.html#downward.experiment.FastDownwardExperiment "downward.experiment.FastDownwardExperiment") class for whole-planner experiments.

* Deprecate `downward.experiments.DownwardExperiment` class.

* Repeat headers between domains in [`downward.reports.taskwise.TaskwiseReport`](index.html#downward.reports.taskwise.TaskwiseReport "downward.reports.taskwise.TaskwiseReport").

### v1.8 (2015-10-02)[¶](#v1-8-2015-10-02 "Permalink to this headline")

#### Lab[¶](#id55 "Permalink to this headline")

* Deprecate predefined experiment steps (`remove\_exp\_dir`,

`zip\_exp\_dir`, `unzip\_exp\_dir`).

* Docs: add FAQs, update docs.

* Add more regression and style tests.

#### Downward Lab[¶](#id56 "Permalink to this headline")

* Parse both evaluated states (evaluated) and evaluations (evaluations).

* Add example experiment showing how to make reports for data obtained without Lab.

* Add suite\_sat\_strips().

* Parse negative initial h values.

* Support CMake builds.

### v1.7 (2015-08-19)[¶](#v1-7-2015-08-19 "Permalink to this headline")

#### Lab[¶](#id57 "Permalink to this headline")

* Automatically determine whether to queue steps sequentially on the grid.

* Reports: right-align headers (except the left-most one).

* Reports: let `lab.reports.gm()` return 0 if any of the numbers is 0.

* Add test that checks for dead code with vulture.

* Remove Step.remove\_exp\_dir step.

* Remove default time and memory limits for commands. You can now pass

`mem\_limit=None` and `time\_limit=None` to disable limits for a

command.

* Pass `extra\_options` kwarg to

`lab.environments.OracleGridEngineEnvironment` to set

additional options like parallel environments.

* Sort `properties` files by keys.

#### Downward Lab[¶](#id58 "Permalink to this headline")

* Add support for new python driver script `fast-downward.py`.

* Use booktabs package for latex tables.

* Remove vertical lines from Latex tables (recommended by booktabs docs).

* Capitalize attribute names and remove underscores for Latex reports.

* Allow fractional plan costs.

* Set search\_time and total\_time to 0.01 instead of 0.1 if they are 0.0 in the log.

* Parse initial h-value for aborted searches (Florian).

* Use EXIT\_UNSOLVABLE instead of logs to determine unsolvability.

Currently, this exit code is only returned by EHC.

* Exit with warning if search parser is not executable.

* Deprecate `downward/configs.py` module.

* Deprecate `examples/standard\_exp.py` module.

* Remove `preprocess-all.py` script.

* By default, use all CPUs for compiling Fast Downward.

### v1.6[¶](#v1-6 "Permalink to this headline")

#### Lab[¶](#id59 "Permalink to this headline")

* Restore earlier default behavior for grid jobs by passing all environment variables (e.g. `PYTHONPATH`) to the job environments.

#### Downward Lab[¶](#id60 "Permalink to this headline")

* Use write-once revision cache: instead of *cloning* the full FD repo

into the revision cache only *copy* the `src` directory. This

greatly reduces the time and space needed to cache revisions. As a

consequence you cannot specify the destination for the clone

anymore (the `dest` keyword argument is removed from the

`Translator`, `Preprocessor` and `Planner` classes) and only

local FD repositories are supported (see

`downward.checkouts.HgCheckout`). After the files have been

copied into the cache and FD has been compiled, a special file

(`build\_successful`) is written in the cache directory. When

the cached revision is requested later an error is shown if this

file is missing.

* Only use exit codes to reason about error reasons. Merge from FD main if your FD version does not produce meaningful exit codes.

* Preprocess parser: only parse logs and never output files.

* Never copy `all.groups` and `test.groups` files. Old Fast Downward branches need to merge from main.

* Always compress `output.sas` (also for `compact=False`). Use `xz` for compressing.

### v1.5[¶](#v1-5 "Permalink to this headline")

#### Lab[¶](#id61 "Permalink to this headline")

* Add `Experiment.add\_fetcher()` method.

* If all columns have the same value in an uncolored table row, make all values bold, not grey.

* In `Experiment.add\_resource()` and `Run.add\_resource()` set `dest=None` if you don’t want to copy or link the resource, but only need an alias to reference it in a command.

* Write and keep all logfiles only if they actually have content.

* Don’t log time and memory consumption of process groups. It is still an unexplained error if too much wall-clock time is used.

* Randomize task order for grid experiments by default. Use `randomize\_task\_order=False` to disable this.

* Save wall-clock times in properties file.

* Do not replace underscores by dashes in table headers. Instead allow browsers to break lines after underscores.

* Left-justify string and list values in tables.

#### Downward Lab[¶](#id62 "Permalink to this headline")

* Add optional *nick* parameter to Translator, Preprocessor and Planner classes. It defaults to the revision name *rev*.

* Save `hg id` output for each checkout and include it in reports.

* Add *timeout* parameter to `DownwardExperiment.add\_config()`.

* Count malformed-logs as unexplained errors.

* Pass `legend\_location=None` if you don’t need a legend in your plot.

* Pass custom benchmark directories in `DownwardExperiment.add\_suite()` by using the *benchmarks\_dir* keyword argument.

* Do not copy logs from preprocess runs into search runs.

* Reference preprocessed files in run scripts instead of creating links if `compact=True` is given in the experiment constructor (default).

* Remove `unexplained\_error` attribute. Errors are unexplained if `run['error']` starts with ‘unexplained’.

* Remove `\*\_error` attributes. It is preferrable to inspect `\*\_returncode` attributes instead (e.g. `search\_returncode`).

* Make report generation faster (10-fold speedup for big reports).

* Add `DownwardExperiment.add\_search\_parser()` method.

* Run `make clean` in revision-cache after compiling preprocessor and search code.

* Strip executables after compilation in revision-cache.

* Do not copy Lab into experiment directories and grid-steps. Use the global Lab version instead.

### v1.4[¶](#v1-4 "Permalink to this headline")

#### Lab[¶](#id63 "Permalink to this headline")

* Add [`exp.add\_report()`](index.html#lab.experiment.Experiment.add_report "lab.experiment.Experiment.add_report") method to simplify adding reports.

* Use simplejson when available to make loading properties more than twice as fast.

* Raise default check-interval in Calls to 5s. This should reduce Lab’s overhead.

* Send mail when grid experiment finishes. Usage: `MaiaEnvironment(email='mymail@example.com')`.

* Remove `steps.Step.publish\_reports()` method.

* Allow creating nested new files in experiment directory (e.g. `exp.add\_new\_file('path/to/file.txt')`).

* Remove duplicate attributes from reports.

* Make commandline parser available globally as [`lab.experiment.ARGPARSER`](index.html#lab.experiment.ARGPARSER "lab.experiment.ARGPARSER") so users can add custom arguments.

* Add `cache\_dir` parameter in [`Experiment`](index.html#lab.experiment.Experiment "lab.experiment.Experiment") for specifying where Lab stores temporary data.

#### Downward Lab[¶](#id64 "Permalink to this headline")

* Move `downward.experiment.DownwardExperiment` to `downward.experiments.DownwardExperiment`, but keep both import locations indefinitely.

* Flag invalid plans in absolute reports.

* PlanningReport: When you append ‘\_relative’ to an attribute, you will get a table containing the attribute’s values of each configuration relative to the leftmost column.

* Use bzip2 for compressing output.sas files instead of tar+gzip to save space and make opening the files easier.

* Use bzip2 instead of gzip for compressing experiment directories to save space.

* Color absolute reports by default.

* Use log-scale instead of symlog-scale for plots. This produces equidistant grid lines.

* By default place legend right of scatter plots.

* Remove `--dereference` option from tar command.

* Copy (instead of linking) PDDL files into preprocessed-tasks dir.

* Add table with Fast Downward commandline strings and revisions to AbsoluteReport.

### v1.3[¶](#v1-3 "Permalink to this headline")

#### Lab[¶](#id65 "Permalink to this headline")

* For Latex tables only keep the first two and last two hlines.

#### Downward Lab[¶](#id66 "Permalink to this headline")

* Plots: Make category\_styles a dictionary mapping from names to dictionaries of

matplotlib plotting parameters to allow for more and simpler customization.

This means e.g. that you can now change the line style in plots.

* Produce a combined domain- and problem-wise AbsoluteReport if `resolution=combined`.

* Include info in AbsoluteReport if a table has no entries.

* Plots: Add `params` argument for specifying matplotlib parameters like

font-family, label sizes, line width, etc.

* AbsoluteReport: If a non-numerical attribute is included in a domain-wise

report, include some info in the table instead of aborting.

* Add [`Attribute`](index.html#lab.reports.Attribute "lab.reports.Attribute") class for wrapping custom

attributes that need non-default report options and aggregation functions.

* Parse `expansions\_until\_last\_jump` attribute.

* Tex reports: Add number of tasks per domain with new `\numtasks{x}` command

that can be cutomized in the exported texts.

* Add pgfplots backend for plots.

### v1.2[¶](#v1-2 "Permalink to this headline")

#### Lab[¶](#id67 "Permalink to this headline")

* Fetcher: Only copy the link not the content for symbolic links.

* Make properties files more compact by using an indent of 2 instead of 4.

* Nicer format for commandline help for experiments.

* Reports: Only print available attributes if none have been set.

* Fetcher: Pass custom parsers to fetcher to parse values from a finished experiment.

* For geometric mean calculation substitute 0.1 for values <= 0.

* Only show warning if not all attributes for the report are found in the evaluation dir,

don’t abort if at least one attribute is found.

* If an attribute is None for all runs, do not conclude it is not numeric.

* Abort if experiment path contains a colon.

* Abort with warning if all runs have been filtered for a report.

* Reports: Allow specifying a *single* attribute as a string instead of

a list of one string (e.g. attributes=’coverage’).

#### Downward Lab[¶](#id68 "Permalink to this headline")

* If compact=True for a DownwardExperiment, link to the benchmarks instead of copying them.

* Do not call ./build-all script, but build components only if needed.

* Fetch and compile sources only when needed: Only prepare translator and

preprocessor for preprocessing experiments and only prepare planners for

search experiments. Do it in a grid job if possible.

* Save space by deleting the benchmarks directories and omitting the search

directory and validator for preprocess experiments.

* Only support using ‘src’ directory, not the old ‘downward’ dir.

* Use `downward` script regardless of other binaries found in the search directory.

* Do not try to set parent-revision property. It cannot be determined without

fetching the code first.

* Make ProblemPlotReport class more general by allowing the get\_points() method

to return an arbitrary number of points and categories.

* Specify xscale and yscale (linear, log, symlog) in PlotReports.

* Fix removing downward.tmp.\* files (use bash for globbing). This greatly reduces

the needed space for an experiment.

* Label axes in ProblemPlots with `xlabel` and `ylabel`.

* If a grid environment is selected, use all CPUs for compiling Fast Downward.

* Do not use the same plot style again if it has already been assigned by the user.

* Only write plot if valid points have been added.

* DownwardExperiment: Add member `include\_preprocess\_results\_in\_search\_runs`.

* Colored reports: If all configs have the same value in a row and some are None,

highlight the values in green instead of making them grey.

* Never set ‘error’ to ‘none’ if ‘search\_error’ is true.

* PlotReport: Add `legend\_location` parameter.

* Plots: Sort coordinates by x-value for correct connections between points.

* Plots: Filter duplicate coordinates for nicer drawing.

* Use less padding for linear scatterplots.

* Scatterplots: Add `show\_missing` parameter.

* Absolute reports: For absolute attributes (e.g. coverage)

print a list of numbers of problems behind the domain name if not all configs

have a value for the same number of problems.

* Make ‘unsolvable’ an absolute attribute, i.e. one where we consider problem

runs for which not all configs have a value.

* If a non-numeric attribute is present in a domain-wise report, state its type

in the error message.

* Let plots use the `format` parameter given in constructor.

* Allow generation of pgf plot files (only available in matplotlib 1.2).

* Allow generation of pdf and eps plots.

* DownwardReport: Allow passing a single function for `derived\_properties`.

* Plots: Remove code that sets parameters explicitly, sort items in legend.

* Add parameters to PlotReport that set the axes’ limits.

* Add more items to Downward Lab FAQ.

### v1.1[¶](#v1-1 "Permalink to this headline")

#### Lab[¶](#id69 "Permalink to this headline")

* Add filter shortcuts: `filter\_config\_nick=['lama', 'hcea'], filter\_domain=['depot']` (see [`Report`](index.html#lab.reports.Report "lab.reports.Report")) (Florian)

* Ability to use more than one filter function (Florian)

* Pass an optional filter to `Fetcher` to fetch only a subset of results (Florian)

* Better handling of timeouts and memory-outs (Florian)

* Try to guess error reason when run was killed because of resource limits (Florian)

* Do not abort after failed commands by default

* Grid: When –all is passed only run all steps if none are supplied

* Environments: Support Uni Basel maia cluster (Malte)

* Add “pi” example

* Add example showing how to parse custom attributes

* Do not add resources and files again if they are already added to the experiment

* Abort if no runs have been added to the experiment

* Round all float values for the tables

* Add function `lab.tools.sendmail()` for sending e-mails

* Many bugfixes

* Added more tests

* Improved documentation

#### Downward Lab[¶](#id70 "Permalink to this headline")

* Make the files output.sas, domain.pddl and problem.pddl optional for search experiments

* Use more compact table of contents for AbsoluteReports

* Use named anchors in AbsoluteReport (`report.html#expansions`, `report.html#expansions-gripper`)

* Add colored absolute tables (see [`AbsoluteReport`](index.html#downward.reports.absolute.AbsoluteReport "downward.reports.absolute.AbsoluteReport"))

* Do not add summary functions in problem-wise reports

* New report class `ProblemPlotReport`

* Save more properties about experiments in the experiments’s properties file for easy lookup (suite, configs, portfolios, etc.)

* Use separate table for each domain in problem-wise reports

* Sort table columns based on given config filters if given (Florian)

* Do not add VAL source files to experiment

* Parse number of reopened states

* Remove temporary Fast Downward files even if planner was killed

* Divide scatter-plot points into categories and lable them (see [`ScatterPlotReport`](index.html#downward.reports.scatter.ScatterPlotReport "downward.reports.scatter.ScatterPlotReport")) (Florian)

* Only add a highlighting and summary functions for numeric attributes in AbsoluteReports

* Compile validator if it isn’t compiled already

* Downward suites: Allow writing SUITE\_NAME\_FIRST to run the first instance of all domains in SUITE\_NAME

* LocalEnvironment: If `processes` is given, use as many jobs to compile the planner in parallel

* Check python version before creating preprocess experiment

* Add avg, min, max and stddev rows to relative reports

* Add RelativeReport

* Add `DownwardExperiment.set\_path\_to\_python()`

* Many bugfixes

* Improved documentation

