Merlin Documentation
                Release 1.11.0
 The Merlin Development Team
                     Oct 09, 2023
                                                                                                   CONTENTS
1 Merlin Overview                                                                                                    3
  1.1 Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
  1.2 Getting Started . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  57
  1.3 FAQ . . . . . . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  60
  1.4 Command line . . . . .     . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  69
  1.5 Workflows . . . . . . .    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  76
  1.6 Workflow Specification     . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  77
  1.7 Configuration . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  84
  1.8 Variables . . . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  90
  1.9 Merlin Server . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  98
  1.10 Celery . . . . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
  1.11 Virtual environments . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
  1.12 Spack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
  1.13 Contributing . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
  1.14 Docker . . . . . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
                                                                                                                     i
ii
                                                                        Merlin Documentation, Release 1.11.0
Merlin is a tool for running machine learning based workflows. The goal of Merlin is to make it easy to build, run, and
process the kinds of large scale HPC workflows needed for cognitive simulation.
CONTENTS                                                                                                              1
Merlin Documentation, Release 1.11.0
2                                    CONTENTS
                                                                                                             CHAPTER
                                                                                                                  ONE
                                                                                       MERLIN OVERVIEW
Merlin is a distributed task queuing system, designed to allow complex HPC workflows to scale to large numbers of
simulations (we’ve done 100 Million on the Sierra Supercomputer).
Why would you want to run that many simulations? To become your own Big Data generator.
Data sets of this size can be large enough to train deep neural networks that can mimic your HPC application, to be
used for such things as design optimization, uncertainty quantification and statistical experimental inference. Merlin’s
been used to study inertial confinement fusion, extreme ultraviolet light generation, structural mechanics and atomic
physics, to name a few.
How does it work?
In essence, Merlin coordinates complex workflows through a persistent external queue server that lives outside of your
HPC systems, but that can talk to nodes on your cluster(s). As jobs spin up across your ecosystem, workers on those
allocations pull work from a central server, which coordinates the task dependencies for your workflow. Since this
coordination is done via direct connections to the workers (i.e. not through a file system), your workflow can scale to
very large numbers of workers, which means a very large number of simulations with very little overhead.
Furthermore, since the workers pull their instructions from the central server, you can do a lot of other neat things, like
having multiple batch allocations contribute to the same work (think surge computing), or specialize workers to different
machines (think CPU workers for your application and GPU workers that train your neural network). Another neat
feature is that these workers can add more work back to central server, which enables a variety of dynamic workflows,
such as may be necessary for the intelligent sampling of design spaces or reinforcement learning tasks.
Merlin does all of this by leveraging some key HPC and cloud computing technologies, building off open source
components. It uses maestro to provide an interface for describing workflows, as well as for defining workflow task
dependencies. It translates those dependencies into concrete tasks via celery, which can be configured for a variety of
backend technologies (rabbitmq and redis are currently supported). Although not a hard dependency, we encourage
the use of flux for interfacing with HPC batch systems, since it can scale to a very large number of jobs.
The integrated system looks a little something like this:
                                                                                                                          3
Merlin Documentation, Release 1.11.0
For more details, check out the rest of the documentation.
Need help? merlin@llnl.gov
1.1 Tutorial
Estimated time
     • 3 hours
Grab your laptop and coffee, and dive into this 7-module tutorial to become a Merlin expert.
This hands-on tutorial introduces Merlin through some example workflows. In it, you will install Merlin on your local
machine, stand up a virtual server and run both a simple workflow and a quasi-real-life physicsy simulation that couples
a physics application with visualization and machine learning.
You’ll also learn how to use some advanced features and help make Merlin better. Finally we offer some tips and tricks
for porting and scaling up your application.
1.1.1 0. Before you start
It will be helpful to have these steps already completed before you start the tutorial modules:
     • Make sure you have python 3.6 or newer.
     • Make sure you have pip version 22.3 or newer.
           • You can upgrade pip to the latest version with:
       pip install --upgrade pip
4                                                                                       Chapter 1. Merlin Overview
                                                                       Merlin Documentation, Release 1.11.0
         • OR you can upgrade to a specific version with:
      pip install --upgrade pip==x.y.z
    • Make sure you have GNU make tools and compilers.
    • (OPTIONAL) Install docker.
         • Download OpenFOAM image with:
      docker pull cfdengine/openfoam
         • Download redis image with:
      docker pull redis
1.1.2 Introduction
This module introduces you to Merlin, some of the technology behind it, and how it works.
Prerequisites
    • Curiosity
Estimated time
    • 20 minutes
You will learn
    • What Merlin is and why you might consider it
    • Why it was built and what are some target use cases
    • How it is designed and what the underlying tech is
 Table of Contents:
     • What is Merlin?
     • Why Merlin? What’s the need?
     • How can Merlin run so many simulations?
     • So what exactly does Merlin do?
     • How is it designed?
     • What is in this Tutorial?
1.1. Tutorial                                                                                             5
Merlin Documentation, Release 1.11.0
What is Merlin?
Summary
Merlin is a toolkit designed to enable HPC-focused simulation workflows with distributed cloud compute technologies.
This helps simulation workflows push to immense scale. (Like 100 million.)
At its core, Merlin translates a text-based, command-line focused workflow description into a set of discrete tasks.
These tasks live on a centralized broker (e.g. a separate server) that persists outside of your HPC batch allocation.
Autonomous workers in different allocations (even on different machines) can then connect to this server, pull off and
execute these tasks asynchronously.
Why Merlin? What’s the need?
That sounds complicated. Why would you care to do this?
The short answer: machine learning
The longer answer: machine learning and data science are becoming an integral part of scientific inquiry. The problem
is that machine learning models are data hungry: it takes lots and lots of simulations to train machine learning models
on their outputs. Unfortunately HPC systems were designed to execute a few large hero simulations, not many smaller
simulations. Naively pushing standard HPC workflow tools to hundreds of thousands and millions of simulations can
lead to some serious problems.
Workflows, applications and machines are becoming more complex, but subject matter experts need to devote time and
attention to their applications and often require fine command-line level control. Furthermore, they rarely have the time
to devote to learning workflow systems.
With the expansion of data-driven computing, the HPC scientist needs to be able to run more simulations through
complex multi-component workflows.
Merlin targets HPC workflows that require many simulations. These include:
                                            Table 1: Merlin Targeted Use Cases
  Emulator building              Running enough simulations to build an emulator (or “surrogate model”) of an expen-
                                 sive computer code, such as needed for uncertainty quantification
  Iterative sampling             Executing some simulations and then choosing new ones to run based on the results
                                 obtained thus far
  Active learning                Iteratively sampling coupled with emulator building to efficiently train a machine
                                 learning model
  Design optimization            Using a computer code to optimize a model design, perhaps robustly or under uncer-
                                 tainty
  Reinforcement learning         Building a machine learning model by subsequently exposing it to lots of trials, giving
                                 it a reward/penalty for the outcomes of those trials
  Hierarchical simulation        Running low-fidelity simulations to inform which higher fidelity simulations to execute
  Heterogeneous workflows        Workflows that require different steps to execute on different hardware and/or systems
Many scientific and engineering problems require running lots of simulations. But accomplishing these tasks effectively
in an unstable bleeding edge HPC environment can be dicey. The tricks that work for 100 simulations won’t work for
10 thousand, let alone 100 million.
We made Merlin to make high-frequency extreme scale computing easy.
6                                                                                       Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
How can Merlin run so many simulations?
The good news is that distributed cloud compute technology has really pushed the frontier of scalability. Merlin helps
bring this tech to traditional scientific HPC.
Traditionally, HPC workflow systems tie workflow steps to HPC resources and coordinate the execution of tasks and
management of resources one of two ways:
                                     Table 2: Traditional HPC Workflow Philosophies
  External                                     Coordination       • Separate batch jobs for each task
                                                                  • External daemon tracks dependencies and jobs
                                                                  • Progress monitored with periodic polling (of files
                                                                    or batch system)
  Internal Coordination                                           • Multiple tasks bundled into larger batch jobs
                                                                  • Internal daemon tracks dependencies and re-
                                                                    sources
                                                                  • Progress monitored via polling (of filesystem or
                                                                    message passing)
External coordination ties together independent batch jobs each executing workflow sub-tasks with an external mon-
itor. This monitor could be a daemon or human that monitors either the batch or file system via periodic polling and
orchestrates task launch dependencies.
External coordination can tailor the resources to the task, but cannot easily run lots of concurrent simulations (since
batch systems usually limit the number of jobs a user can queue at once).
Internal coordination puts the monitor within a larger batch job that allocates resources inside that job for the specific
tasks at hand.
Internal coordination can run many more concurrent tasks by bundling smaller jobs into larger jobs, but cannot tailor
the resources to the task at hand. This precludes workflows that, for instance, require one step on CPU hardware and
another on a GPU machine.
Instead of tying resources to tasks, Merlin does this:
1.1. Tutorial                                                                                                            7
Merlin Documentation, Release 1.11.0
                                        Table 3: Merlin’s Workflow Philosophy
  Centralized Coordination of Producers & Consumers              • Batch jobs and workers decoupled from tasks
                                                                 • Centralized queues visible to multiple jobs
                                                                 • Progress and dependencies handled via direct
                                                                   worker connections to central message server and
                                                                   results database
Merlin decouples workflow tasks from workflow resources.
Merlin avoids a command-and-control approach to HPC resource management for a workflow. Instead of having the
workflow coordinator ask for and manage HPC resources and tasks, the Merlin coordinator just manages tasks. Task-
agnostic resources can then independently connect (and disconnect) to the coordinator.
In Merlin, this producer-consumer workflow happens through two commands:
merlin run <workflow file> (producer)
and
merlin run-worker <workflow file> (consumer).
The merlin run command populates the central queue(s) with work to do and the merlin run-worker command
drains the queue(s) by executing the task instructions. Each new instance of merlin run-worker creates a new
consumer. These consumers can exist on different machines in different batch allocations, anywhere that can see the
central server. Likewise merlin run can populate the queue from any system that can see the queue server, including
other workers. In principle, this means a researcher can push new work onto an already running batch allocation of
workers, or re-direct running jobs to work on higher-priority work.
The benefits of producer-consumer workflows
The increased flexibility that comes from decoupling what HPC applications you run from where you run them can be
extremely enabling.
Merlin allows you to
    • Scale to very large number of simulations by avoiding common HPC bottlenecks
    • Automatically take advantage of free nodes to process your workflow faster
    • Create iterative workflows, like as needed for active machine learning
    • Dynamically add more tasks to already-running jobs
    • Have cross-machine and cross-batch-job workflows, with different steps executing on different resources, but still
      coordinated
The producer-consumer approach to workflows allows for increased flexibility and scalability. For this reason it has
become a mainstay of cloud-compute microservices, which allow for extremely distributed asynchronous computing.
8                                                                                   Chapter 1. Merlin Overview
                                                                             Merlin Documentation, Release 1.11.0
Many asynchronous task and workflow systems exist, but the majority are focused around this microservices model,
where a system is set up (and managed) by experts that build a single workflow. This static workflow gets tested and
hardened and exists as a service for their users (e.g. an event on a website triggers a discrete set of tasks). HPC, and
in particular scientific HPC brings its own set of challenges that make a direct application of microservices to HPC
workflows challenging.
                         Table 4: Challenges for bringing microservices to scientific HPC Work-
                         flows
  Challenge                                                    Requirement
  Workflows can change from day-to-day as researchers          Workflows need to be dynamic, not static.
  explore new simulations, configurations, and questions.
  Workflow components are usually different executables,       Workflows need to intuitively support multiple lan-
  pre- and post-processing scripts and data aggregation        guages.
  steps written in different languages.
  These components often need command-line-level con-          Workflows need to support shell syntax and environment
  trol of task instructions.                                   variables.
  Components frequently require calls to a batch system        Workflows need a natural way to launch parallel jobs
  scheduler for parallel job execution.                        that use more resources then a single worker.
  Tasks can independently create large quantities of data.     Dataflow models could be bottlenecks. Workflows should
                                                               take advantage of parallel file systems.
  HPC systems (in particular leadership class machines)        Workflows need to be able to restart, retry and rerun
  can experience unforeseen outages.                           failed steps without needing to run the entire workflow.
Merlin was built specifically to address the challenges of porting microservices to HPC simulations.
So what exactly does Merlin do?
Merlin wraps a heavily tested and well used asynchronous task queuing library in a skin and syntax that is natural
for HPC simulations. In essence, we extend maestro by hooking it up to celery. We leverage maestro’s HPC-friendly
workflow description language and translate it to discrete celery tasks.
Why not just plain celery?
Celery is extremely powerful, but this power can be a barrier for many science and engineering subject matter experts,
who might not be python coders. While this may not be an issue for web developers, it presents a serious challenge
to many scientists who are used to running their code from a shell command line. By wrapping celery commands in
maestro steps, we not only create a familiar environment for users (since maestro steps look like shell commands), but
we also create structure around celery dependencies. Maestro also has interfaces to common batch schedulers (e.g.
slurm and flux)[*]_ for parallel job control.
So why Merlin and not just plain maestro?
The main reason: to run lots of simulations for machine learning applications. Basically Merlin scales maestro.
Maestro follows an external coordinator model. Maestro workflow DAGs (directed acyclic graphs) need to be unrolled
(concretized) ahead of time, so that batch dependencies can be calculated and managed. This graph problem becomes
very expensive as the number of tasks approaches a few hundred. (Not to mention most batch systems will prevent
a user from queuing more than a few hundred concurrent batch jobs.) In other words, using maestro alone to run
thousands of simulations is not practical.
But with celery, we can dynamically create additional tasks. This means that the DAG can get unrolled by the very
same workers that will execute the tasks, offering a natural parallelism (i.e. much less waiting before starting the work).
What does this mean in practice?
Merlin can quickly queue a lot of simulations.
1.1. Tutorial                                                                                                            9
Merlin Documentation, Release 1.11.0
How quickly? The figure below shows task queuing rates when pushing a simple workflow on the Quartz Supercomputer
to 40 million samples. This measures how quickly simulation ensembles of various sample sizes can get enqueued.
As you can see, by exploiting celery’s dynamic task queuing (tasks that create tasks), Merlin can enqueue hundreds of
thousands of simulations per second. These jobs can then be consumed in parallel, at a rate that depends on the number
of workers you have.
Furthermore, this ability to dynamically add tasks to the queue means that workflows can become more flexible and
responsive. A worker executing a step can launch additional workflows without having to stand up resources to execute
and monitor the execution of those additional steps.
The only downside to being able to enqueue work this quickly is the inability of batch schedulers to keep up. This is
why we recommend pairing Merlin with flux, which results in a scalable but easy-to-use workflow system:
    • Maestro describes the workflow tasks
    • Merlin orchestrates the task executions
    • Flux schedules the HPC resources
Here’s an example of how Merlin, maestro and flux can all work together to launch a workflow on multiple machines.
10                                                                                    Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
The scientist describes her workflow with a maestro-like <workflow file>. Her workflow consists of two steps:
   1. Run many parallel CPU-only jobs, varying her simulation parameters of interest
   2. Use a GPU to train a deep learning model on the results of those simulations
She then types merlin run <workflow file>, which translates that maestro file into celery commands and sends
those tasks to two separate queues on a centralized server (one for CPU work and one for GPU work).
She then launches a batch allocation on the CPU machine, which contains the command merlin run-workers
<workflow file> --steps 1. Workers start up under flux, pull work from the server’s CPU queue and call flux to
launch the parallel simulations asynchronously.
She also launches a separate batch request on the GPU machine with merlin run-workers <workflow file>
--steps 2. These workers connect to the central queue associated with the GPU step.
When the simulations in step 1 finish, step 2 will automatically start. In this fashion, Merlin allows the scientist to
coordinate a highly scalable asynchronous multi-machine heterogeneous workflow.
This is of course a simple example, but it does show how the producer-consumer philosophy in HPC workflows can
be quite enabling. Merlin’s goal is to make it easy for HPC-focused subject matter experts to take advantage of the
advances in cloud computing.
How is it designed?
Merlin leverages a number of open source technologies, developed and battle-hardened in the world of distributed
computing. We decided to do this instead of having to build, test and maintain stand-alone customized (probably
buggy) versions of software that will probably not be as fully featured.
There are differing philosophies on how much third-party software to rely upon. On the one hand, building our system
off ubiquitous open source message passing libraries increases the confidence in our software stack’s performance,
especially at scale (for instance, celery is robust enough to keep Instagram running). However, doing so means that
when something breaks deep down, it can be difficult to fix (if at all). Indeed if there’s an underlying “feature” that
we’d like to work around, we could be stuck. Furthermore, the complexity of the software stack can be quite large,
1.1. Tutorial                                                                                                       11
Merlin Documentation, Release 1.11.0
such that our team couldn’t possibly keep track of it all. These are valid concerns; however, we’ve found it much
easier to quickly develop a portable system with a small team by treating (appropriately chosen) third party libraries as
underlying infrastructure. (Sure you could build and use your own compiler, but should you?)
Merlin manages the increased risk that comes with relying on software that is out of our control by:
    1. Building modular software that can easily be reconfigured / swapped for other tech
    2. Participating as developers for those third-party packages upon which rely (for instance we often kick enhance-
       ments and bug fixes to maestro)
    3. Using continuous integration and testing to catch more errors as they occur
This section talks about some of those underlying technologies, what they are, and why they were chosen.
A brief technical dive into some underlying tech
Merlin extends maestro with celery, which in turn can be configured to interface with a variety of message queue
brokers and results backends. In practice, we like to use RabbitMQ and Redis for our broker and backend respectively,
because of their features and reliability, especially at scale.
                                                Table 5: Key Merlin Tech Components
 Component                            Reasoning
 maestro                              shell-like workflow descriptions, batch system interfaces
 celery                               highly scalable, supports multiple brokers and backends
 RabbitMQ                             resilience, support for multiple users and queues
 Redis                                database speed, scalability
 cryptography                         secure Redis results
 flux (optional)                      portability and scalability of HPC resource allocation
The different components interact to populate and drain the message queue broker of workflow tasks.
When a call is made to merlin run, maestro turns the workflow description (composed of “steps” with “parameters”
and “samples”) into a task dependency graph. Merlin translates this graph into discrete celery task commands*0
   0 The flux and slurm interfaces used by Merlin differ from the versions bundled with maestro to decouple job launching from batch submission.
12                                                                                                      Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
Calls to merlin run-worker cause celery workers to connect to both the message broker and results database. The
workers pull tasks from the broker and begin to execute the instructions therein. When finished, a worker posts the
results (task status metadata, such as “SUCCESS” or “FAIL”) to the results database and automatically grabs another
task from the queue. When additional workers come along (through other explicit calls to merlin run-worker), they
connect to the broker and help out with the workflow.
Multiple vs. Single Queues
RabbitMQ brokers can have multiple distinct queues. To take advantage of this feature, Merlin lets you assign workflow
steps and workers to different queues. (Steps must be assigned to a single queue, but workers can connect to multiple
queues at once.) The advantage of a single queue is simplicity, both in workflow design and scalability. However, having
multiple queues allows for prioritization of work (the express checkout lane at the grocery store) and customization of
workers (specialized assembly line workers tailored for a specific task).
What is in this Tutorial?
This tutorial will show you how to:
    • Install Merlin and test that it works correctly
    • Build a basic workflow and scale it up, introducing you to Merlin’s syntax and how it differs from maestro.
    • Run a “real” physics simulation based workflow, with post-processing of results, visualization and machine
       learning.
    • Use Merlin’s advanced features to do things like interface with batch systems, distribute a workflow across
       machines and dynamically add new samples to a running workflow.
    • Contribute to Merlin, through code enhancements and bug reports.
    • Port your own application, with tips and tricks for building and scaling up workflows.
1.1.3 Installation
Prerequisites
    • shell (bash, csh, etc, if running on Windows, use a linux container)
    • python3 >= python3.6
    • pip3
    • wget
    • build tools (make, C/C++ compiler)
    • (OPTIONAL) docker (required for Module 4: Run a Real Simulation)
    • (OPTIONAL) file editor for docker config file editing
Estimated time
    • 20 minutes
You will learn
    • How to install merlin in a virtual environment using pip.
1.1. Tutorial                                                                                                         13
Merlin Documentation, Release 1.11.0
     • How to install a container platform eg. singularity, docker, or podman.
     • How to configure merlin.
     • How to test/verify the installation.
  Table of Contents:
      • Installing Merlin
           – Redis Server
      • Configuring Merlin
      • Checking/Verifying Installation
      • (OPTIONAL) Docker Advanced Installation
           – RabbitMQ Server
           – Redis TLS Server
This section details the steps necessary to install merlin and its dependencies. Merlin will then be configured for the
local machine and the configuration will be checked to ensure a proper installation.
Installing Merlin
A merlin installation is required for the subsequent modules of this tutorial.
Once merlin is installed, it requires servers to operate. While you are able to host your own servers, we will use merlin’s
containerized servers in this tutorial. However, if you prefer to host your own servers you can host a redis server that
is accessible to your current machine. Your computer/organization may already have a redis server available you can
use, please check with your local system administrator.
Create a virtualenv using python3 to install merlin.
python3 -m venv --prompt merlin merlin_venv
Activate the virtualenv.
source merlin_venv/bin/activate
or
source merlin_venv/bin/activate.csh
The (merlin) <shell prompt> will appear after activating.
You should upgrade pip and setuptools before proceeding.
pip3 install setuptools pip -U
Install merlin through pip.
pip3 install merlin
Check to make sure merlin installed correctly.
which merlin
14                                                                                       Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
You should see that it was installed in your virtualenv, like so:
~/<path_to_virtualenv>/merlin_venv/bin/merlin
If this is not the output you see, you may need to restart your virtualenv and try again.
When you are done with the virtualenv you can deactivate it using deactivate, but leave the virtualenv activated for
the subsequent steps.
deactivate
Redis Server
A redis server is required for the celery results backend server, this same server can also be used for the celery broker.
We will be using merlin’s containerized server however we will need to download one of the supported container
platforms avaliable. For the purpose of this tutorial we will be using singularity.
# Update and install singularity dependencies
apt-get update && apt-get install -y \
   build-essential \
   libssl-dev \
   uuid-dev \
   libgpgme11-dev \
   squashfs-tools \
   libseccomp-dev \
   pkg-config
# Download dependency go
wget https://go.dev/dl/go1.18.1.linux-amd64.tar.gz
# Extract go into local
tar -C /usr/local -xzf go1.18.1.linux-amd64.tar.gz
# Remove go tar file
rm go1.18.1.linux-amd64.tar.gz
# Update PATH to include go
export PATH=$PATH:/usr/local/go/bin
# Download singularity
wget https://github.com/sylabs/singularity/releases/download/v3.9.9/singularity-ce-3.9.9.
 ˓→tar.gz
# Extract singularity
tar -xzf singularity-ce-3.9.9.tar.gz
# Configure and install singularity
cd singularity-ce-3.9.9
./mconfig && \
   make -C ./builddir && \
   sudo make -C ./builddir install
1.1. Tutorial                                                                                                          15
Merlin Documentation, Release 1.11.0
Configuring Merlin
Merlin requires a configuration script for the celery interface. Run this configuration method to create the app.yaml
configuration file.
merlin config --broker redis
The merlin config command above will create a file called app.yaml in the ~/.merlin directory. If you are
running a redis server locally then you are all set, look in the ~/.merlin/app.yaml file to see the configuration, it
should look like the configuration below.
broker:
      name: redis
      server: localhost
      port: 6379
      db_num: 0
results_backend:
      name: redis
      server: localhost
      port: 6379
      db_num: 0
More detailed information on configuring Merlin can be found in the configuration section.
Checking/Verifying Installation
First launch the merlin server containers by using the merlin server commands
merlin server init
merlin server start
A subdirectory called merlin_server/ will have been created in the current run directory. This contains all of the
proper configuration for the server containers merlin creates. Configuration can be done through the merlin server
config command, however users have the flexibility to edit the files directly in the directory. Additionally an precon-
figured app.yaml file has been created in the merlin_server/ subdirectory to utilize the merlin server containers .
To use it locally simply copy it to the run directory with a cp command.
cp ./merlin_server/app.yaml .
You can also make this server container your main server configuration by replacing the one located in your home
directory. Make sure you make back-ups of your current app.yaml file in case you want to use your previous configura-
tions. Note: since merlin servers are created locally on your run directory you are allowed to create multiple instances
of merlin server with their unique configurations for different studies. Simply create different directories for each study
and run merlin server init in each directory to create an instance for each.
mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak
cp ./merlin_server/app.yaml ~/.merlin/
The merlin info command will check that the configuration file is installed correctly, display the server configuration
strings, and check server access.
merlin info
If everything is set up correctly, you should see:
16                                                                                      Chapter 1. Merlin Overview
                                                                         Merlin Documentation, Release 1.11.0
.
.
.
Merlin Configuration
-------------------------
 config_file               |  <user home>/.merlin/app.yaml
 is_debug                  |  False
 merlin_home               |  <user home>/.merlin
 merlin_home_exists        |  True
 broker server             |  redis://localhost:6379/0
 results server            |  redis://localhost:6379/0
Checking server connections:
----------------------------
broker server connection: OK
results server connection: OK
Python Configuration
-------------------------
.
.
.
(OPTIONAL) Docker Advanced Installation
RabbitMQ Server
This optional section details the setup of a rabbitmq server for merlin. A rabbitmq server can be started to provide
the broker, the redis server will still be required for the backend. Merlin is configured to use ssl encryption for all
communication with the rabbitmq server. An ssl server requires ssl certificates to encrypt the communication through
the python ssl module python ssl . This tutorial can use self-signed certificates created by the user for use in the
rabbitmq server. The rabbitmq server uses Transport Layer Security (TLS) (often known as “Secure Sockets Layer”).
Information on rabbitmq with TLS can be found here: rabbit TLS
A set of self-signed keys is created through the tls-gen package. These keys are then copied to a common directory
for use in the rabbitmq server and python.
git clone https://github.com/michaelklishin/tls-gen.git
cd tls-gen/basic
make CN=my-rabbit CLIENT_ALT_NAME=my-rabbit SERVER_ALT_NAME=my-rabbit
make verify
mkdir -p ${HOME}/merlinu/cert_rabbitmq
cp result/* ${HOME}/merlinu/cert_rabbitmq
The rabbitmq docker service can be added to the previous docker-compose.yml file.
version: '3'
networks:
                                                                                                   (continues on next page)
1.1. Tutorial                                                                                                           17
Merlin Documentation, Release 1.11.0
                                                                                            (continued from previous page)
   mernet:
       driver: bridge
services:
   redis:
       image: 'redis:latest'
       container_name: my-redis
       ports:
         - "6379:6379"
       networks:
         - mernet
   rabbitmq:
       image: rabbitmq:3-management
       container_name: my-rabbit
       tty: true
       ports:
         - "15672:15672"
         - "15671:15671"
         - "5672:5672"
         - "5671:5671"
       environment:
         - RABBITMQ_SSL_CACERTFILE=/cert_rabbitmq/ca_certificate.pem
         - RABBITMQ_SSL_KEYFILE=/cert_rabbitmq/server_key.pem
         - RABBITMQ_SSL_CERTFILE=/cert_rabbitmq/server_certificate.pem
         - RABBITMQ_SSL_VERIFY=verify_none
         - RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=false
         - RABBITMQ_DEFAULT_USER=merlinu
         - RABBITMQ_DEFAULT_VHOST=/merlinu
         - RABBITMQ_DEFAULT_PASS=guest
       volumes:
         - ~/merlinu/cert_rabbitmq:/cert_rabbitmq
       networks:
         - mernet
   merlin:
       image: 'llnl/merlin'
       container_name: my-merlin
       tty: true
       volumes:
         - ~/merlinu/:/home/merlinu
       networks:
         - mernet
When running the rabbitmq broker server, the config can be created with the default merlin config command. If you
have already run the previous command then remove the ~/.merlin/app.yaml or ~/merlinu/.merlin/app.yaml
file , and run the merlin config command again.
merlin config
The app.yaml file will need to be edited to add the rabbitmq settings in the broker section of the app.yaml file. The
server: should be changed to my-rabbit. The rabbitmq server will be accessed on the default TLS port, 5671.
18                                                                                   Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
broker:
      name: rabbitmq
      server: my-rabbit
      password: ~/.merlin/rabbit.pass
results_backend:
      name: redis
      server: my-redis
      port: 6379
      db_num: 0
To complete the config create a file ~/merlinu/.merlin/rabbit.pass and add the password guest.
The aliases defined previously can be used with this set of docker containers.
Redis TLS Server
This optional section details the setup of a redis server with TLS for merlin. The reddis TLS configuration can be found
in the Security with redis section. A newer redis (version 6 or greater) must be used to enable TLS.
A set of self-signed keys is created through the tls-gen package. These keys are then copied to a common directory
for use in the redis server and python.
git clone https://github.com/michaelklishin/tls-gen.git
cd tls-gen/basic
make CN=my-redis CLIENT_ALT_NAME=my-redis SERVER_ALT_NAME=my-redis
make verify
mkdir -p ${HOME}/merlinu/cert_redis
cp result/* ${HOME}/merlinu/cert_redis
The configuration below does not use client verification --tls-auth-clients no so the ssl files do not need to be
defined as shown in the Security with redis section.
version: '3'
networks:
   mernet:
      driver: bridge
services:
   redis:
      image: 'redis'
      container_name: my-redis
      command:
         - --port 0
         - --tls-port 6379
         - --tls-ca-cert-file /cert_redis/ca_certificate.pem
         - --tls-key-file /cert_redis/server_key.pem
         - --tls-cert-file /cert_redis/server_certificate.pem
         - --tls-auth-clients no
      ports:
         - "6379:6379"
      volumes:
                                                                                                    (continues on next page)
1.1. Tutorial                                                                                                            19
Merlin Documentation, Release 1.11.0
                                                                                            (continued from previous page)
       - "~/merlinu/cert_redis:/cert_redis"
     networks:
       - mernet
   rabbitmq:
     image: rabbitmq:3-management
     container_name: my-rabbit
     tty: true
     ports:
       - "15672:15672"
       - "15671:15671"
       - "5672:5672"
       - "5671:5671"
     volumes:
       - "~/merlinu/rabbbitmq.conf:/etc/rabbitmq/rabbitmq.conf"
       - "~/merlinu/cert_rabbitmq:/cert_rambbitmq"
     networks:
       - mernet
The rabbitmq.conf file contains the configuration, including ssl, for the rabbitmq server.
default_vhost = /merlinu
default_user = merlinu
default_pass = guest
listeners.ssl.default = 5671
ssl.options.ccertfile = /cert_rabbitmq/ca_certificate.pem
ssl.options.certfile = /cert_rabbitmq/server_certificate.pem
ssl.options.keyfile = /cert_rabbitmq/server_key.pem
ssl.options.verify = verify_none
ssl.options.fail_if_no_peer_cert = false
Once this docker-compose file is run, the merlin app.yaml file is changed to use the redis TLS server rediss instead
of redis.
1.1.4 Hello, World!
This hands-on module walks through the steps of building and running a simple merlin workflow.
Prerequisites
    • Module 2: Installation
Estimated time
    • 30 minutes
You will learn
    • The components of a merlin workflow specification.
    • How to run a simple merlin workflow.
20                                                                                   Chapter 1. Merlin Overview
                                                                              Merlin Documentation, Release 1.11.0
     • How to interpret the results of your workflow.
 Table of Contents:
      • Get Example Files
      • Specification File
            – Section: description
            – Section: global.parameters
            – Section: study
      • Try It!
      • Run Distributed!
      • Using Samples
Get Example Files
merlin example is a command line tool that makes it easy to get a basic workflow up and running. To see a list of
all the examples provided with merlin you can run:
$ merlin example list
For this tutorial we will be using the hello example. Run the following commands:
$ merlin example hello
$ cd hello/
This will create and move into directory called hello, which contains these files:
     • my_hello.yaml – this spec file is partially blank. You will fill in the gaps as you follow this module’s steps.
     • hello.yaml – this is a complete spec without samples. You can always reference it as an example.
     • hello_samples.yaml – same as before, but with samples added.
     • make_samples.py – this is a small python script that generates samples.
     • requirements.txt – this is a text file listing this workflow’s python dependencies.
Specification File
Central to Merlin is something called a specification file, or a “spec” for short. The spec defines all aspects of your
workflow. The spec is formatted in yaml. If you’re unfamiliar with yaml, it’s worth reading up on for a few minutes.
  Warning: Stray whitespace can break yaml; make sure your indentation is consistent.
Let’s build our spec piece by piece. For each spec section listed below, fill in the blank yaml entries of my_hello.yaml
with the given material.
1.1. Tutorial                                                                                                         21
Merlin Documentation, Release 1.11.0
Section: description
Just what it sounds like. Name and briefly summarize your workflow.
description:
      name: hello world workflow
      description: say hello in 2 languages
Section: global.parameters
Global parameters are constants that you want to vary across simulations. Steps that contain a global parameter or
depend on other steps that contain a global parameter are run for each index over parameter values. The label is the
pattern for a filename that will be created for each value.
global.parameters:
      GREET:
           values : ["hello","hola"]
           label : GREET.%%
      WORLD:
           values : ["world","mundo"]
           label : WORLD.%%
Note: %% is a special token that defines where the value in the label is placed. In this case the parameter labels will
be GREET.hello, GREET.hola, etc. The label can take a custom text format, so long as the %% token is included to be
able to substitute the parameter’s value in the appropriate place.
So this will give us 1) an English result, and 2) a Spanish one (you could add as many more languages as you want, as
long as both parameters hold the same number of values).
Section: study
This is where you define workflow steps. While the convention is to list steps as sequentially as possible, the only factor
in determining step order is the dependency directed acyclic graph (DAG) created by the depends field.
study:
      - name: step_1
         description: say hello
         run:
              cmd: echo "$(GREET), $(WORLD)!"
      - name: step_2
         description: print a success message
         run:
              cmd: print("Hurrah, we did it!")
              depends: [step_1]
              shell: /usr/bin/env python3
Note: The - denotes a list item in YAML. To add elements, simply add new elements prefixed with a hyphen
22                                                                                     Chapter 1. Merlin Overview
                                                                        Merlin Documentation, Release 1.11.0
$(GREET) and $(WORLD) expand the global parameters separately into their two values. .. $(step_1.workspace)
gets the path to step_1. The default value for shell is /bin/bash. In step_2 we override this to use python instead.
Steps must be defined as nodes in a DAG, so no cyclical dependencies are allowed. Our step DAG currently looks like
this:
Since our global parameters have 2 values, this is actually what the DAG looks like:
It looks like running step_2 twice is redundant. Instead of doing that, we can collapse it back into a single step, by
having it wait for both parameterized versions of step_1 to finish. Add _* to the end of the step name in step_1’s
depend entry. Go from this:
depends: [step_1]
. . . to this:
depends: [step_1_*]
Now the DAG looks like this:
1.1. Tutorial                                                                                                       23
Merlin Documentation, Release 1.11.0
Your full hello world spec my_hello.yaml should now look like this (an exact match of hello.yaml):
description:
     name: hello
     description: a very simple merlin workflow
global.parameters:
     GREET:
           values : ["hello","hola"]
           label : GREET.%%
     WORLD:
           values : ["world","mundo"]
           label : WORLD.%%
study:
     - name: step_1
        description: say hello
        run:
              cmd: echo "$(GREET), $(WORLD)!"
     - name: step_2
        description: print a success message
        run:
              cmd: print("Hurrah, we did it!")
              depends: [step_1_*]
              shell: /usr/bin/env python3
The order of the spec sections doesn’t matter.
Note: At this point, my_hello.yaml is still maestro-compatible. The primary difference is that maestro won’t un-
derstand anything in the merlin block, which we will still add later. If you want to try it, run: $ maestro run
my_hello.yaml
24                                                                               Chapter 1. Merlin Overview
                                                                        Merlin Documentation, Release 1.11.0
Try It!
First, we’ll run merlin locally. On the command line, run:
$ merlin run --local my_hello.yaml
If your spec is bugless, you should see a few messages proclaiming successful step completion, like this (for now we’ll
ignore the warning):
          *
     *~~~~~
    *~~*~~~*          __ __                  _ _
  /    ~~~~~         | \/ |                 | (_)
       ~~~~~         | \ / | ___ _ __| |_ _ __
      ~~~~~*         | |\/| |/ _ \ '__| | | '_ \
     *~~~~~~~        | | | | __/ | | | | | | |
    ~~~~~~~~~~       |_| |_|\___|_| |_|_|_| |_|
  *~~~~~~~~~~~
     ~~~*~~~*        Machine Learning for HPC Workflows
[2020-02-07 09:35:49: WARNING] Workflow specification missing
  encouraged 'merlin' section! Run 'merlin example' for examples.
Using default configuration with no sampling.
[2020-02-07 09:35:49: INFO] Study workspace is 'hello_20200207-093549'.
[2020-02-07 09:35:49: INFO] Reading app config from file ~/.merlin/app.yaml
[2020-02-07 09:35:49: INFO] Calculating task groupings from DAG.
[2020-02-07 09:35:49: INFO] Converting graph to celery tasks.
[2020-02-07 09:35:49: INFO] Launching tasks.
[2020-02-07 09:35:49: INFO] Executing step 'step1_HELLO.hello' in 'hello_20200207-093549/
 ˓→step1/HELLO.hello'...
[2020-02-07 09:35:54: INFO] Step 'step1_HELLO.hello' in 'hello_20200207-093549/step1/
 ˓→HELLO.hello' finished successfully.
[2020-02-07 09:35:54: INFO] Executing step 'step2_HELLO.hello' in 'hello_20200207-093549/
 ˓→step2/HELLO.hello'...
[2020-02-07 09:35:59: INFO] Step 'step2_HELLO.hello' in 'hello_20200207-093549/step2/
 ˓→HELLO.hello' finished successfully.
[2020-02-07 09:35:59: INFO] Executing step 'step1_HELLO.hola' in 'hello_20200207-093549/
 ˓→step1/HELLO.hola'...
[2020-02-07 09:36:04: INFO] Step 'step1_HELLO.hola' in 'hello_20200207-093549/step1/
 ˓→HELLO.hola' finished successfully.
[2020-02-07 09:36:04: INFO] Executing step 'step2_HELLO.hola' in 'hello_20200207-093549/
 ˓→step2/HELLO.hola'...
[2020-02-07 09:36:09: INFO] Step 'step2_HELLO.hola' in 'hello_20200207-093549/step2/
 ˓→HELLO.hola' finished successfully.
Great! But what happened? We can inspect the output directory to find out.
Look for a directory named hello_<TIMESTAMP>. That’s your output directory. Within, there should be a directory
for each step of the workflow, plus one called merlin_info. The whole file tree looks like this:
1.1. Tutorial                                                                                                      25
Merlin Documentation, Release 1.11.0
A lot of stuff, right? Here’s what it means:
     • The 3 yaml files inside merlin_info/ are called the provenance specs. They are copies of the original spec that
       was run, some showing under-the-hood variable expansions.
     • MERLIN_FINISHED files indicate that the step ran successfully.
     • .sh files contain the command for the step.
     • .out files contain the step’s stdout. Look at one of these, and it should contain your “hello” message.
     • .err files contain the step’s stderr. Hopefully empty, and useful for debugging.
Run Distributed!
Important: Before trying this, make sure you’ve properly set up your merlin config file app.yaml. Run $ merlin
info for information on your merlin configuration.
Now we will run the same workflow, but in parallel on our task server:
$ merlin run my_hello.yaml
If your merlin configuration is set up correctly, you should see something like this:
          *
    *~~~~~
   *~~*~~~*            __   __                _ _
                                                                                                     (continues on next page)
26                                                                                      Chapter 1. Merlin Overview
                                                                    Merlin Documentation, Release 1.11.0
                                                                                        (continued from previous page)
  /    ~~~~~       | \/ |                 | (_)
       ~~~~~       | \ / | ___ _ __| |_ _ __
      ~~~~~*       | |\/| |/ _ \ '__| | | '_ \
     *~~~~~~~      | | | | __/ | | | | | | |
    ~~~~~~~~~~     |_| |_|\___|_| |_|_|_| |_|
  *~~~~~~~~~~~
     ~~~*~~~*      Machine Learning for HPC Workflows
[2020-02-07 13:06:23: WARNING] Workflow specification missing
  encouraged 'merlin' section! Run 'merlin example' for examples.
Using default configuration with no sampling.
[2020-02-07 13:06:23: INFO] Study workspace is 'studies/simple_chain_20200207-130623'.
[2020-02-07 13:06:24: INFO] Reading app config from file ~/.merlin/app.yaml
[2020-02-07 13:06:25: INFO] broker: amqps://user:******@broker:5671//user
[2020-02-07 13:06:25: INFO] backend: redis://user:******@backend:6379/0
[2020-02-07 13:06:25: INFO] Calculating task groupings from DAG.
[2020-02-07 13:06:25: INFO] Converting graph to celery tasks.
[2020-02-07 13:06:25: INFO] Launching tasks.
That means we have launched our tasks! Now we need to launch the workers that will complete those tasks. Run this:
$ merlin run-workers my_hello.yaml
Here’s the expected merlin output message for running workers:
         *
     *~~~~~
    *~~*~~~*         __ __                 _ _
  /    ~~~~~       | \/ |                 | (_)
       ~~~~~       | \ / | ___ _ __| |_ _ __
      ~~~~~*       | |\/| |/ _ \ '__| | | '_ \
     *~~~~~~~      | | | | __/ | | | | | | |
    ~~~~~~~~~~     |_| |_|\___|_| |_|_|_| |_|
  *~~~~~~~~~~~
     ~~~*~~~*      Machine Learning for HPC Workflows
[2020-02-07 13:14:38: INFO] Launching workers from 'hello.yaml'
[2020-02-07 13:14:38: WARNING] Workflow specification missing
  encouraged 'merlin' section! Run 'merlin example' for examples.
Using default configuration with no sampling.
[2020-02-07 13:14:38: INFO] Starting celery workers
[2020-02-07 13:14:38: INFO] ['celery worker -A merlin -n default_worker.%%h -l INFO -Q␣
 ˓→merlin']
Immediately after that, this will pop up:
  -------------- celery@worker_name.%machine770 v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Linux-3.10.0-1062.9.1.1chaos.ch6.x86_64-x86_64-with-redhat-7.7-Maipo␣
                                                                                               (continues on next page)
1.1. Tutorial                                                                                                       27
Merlin Documentation, Release 1.11.0
                                                                                                 (continued from previous page)
 ˓→  2020-02-12 09:53:10
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:                       merlin:0x2aaab20619e8
- ** ---------- .> transport:                 amqps://user:**@server:5671//user
- ** ---------- .> results:                   redis://user:**@server:6379/0
- *** --- * --- .> concurrency:               36 (prefork)
-- ******* ---- .> task events:               OFF (enable -E to monitor tasks in this worker)
--- ***** -----
   -------------- [queues]
                        .> merlin                   exchange=merlin(direct) key=merlin
[tasks]
     . merlin.common.tasks.add_merlin_expanded_chain_to_chord
     . merlin.common.tasks.expand_tasks_with_samples
     . merlin.common.tasks.merlin_step
     . merlin:chordfinisher
     . merlin:queue_merlin_study
[2020-02-12       09:53:11,549:     INFO]     Connected to amqps://user:**@server:5671//user
[2020-02-12       09:53:11,599:     INFO]     mingle: searching for neighbors
[2020-02-12       09:53:12,807:     INFO]     mingle: sync with 2 nodes
[2020-02-12       09:53:12,807:     INFO]     mingle: sync complete
[2020-02-12       09:53:12,835:     INFO]     celery@worker_name.%machine770 ready.
You may not see all of the info logs listed after the Celery C is displayed. If you’d like to see them you can change the
merlin workers’ log levels with the --worker-args tag:
$ merlin run-workers --worker-args "-l INFO" my_hello.yaml
The terminal you ran workers in is now being taken over by Celery, the powerful task queue library that merlin uses
internally. The workers will continue to report their task status here until their tasks are complete.
Workers are persistent, even after work is done. Send a stop signal to all your workers with this command:
$ merlin stop-workers
. . . and a successful worker stop will look like this, with the name of specific worker(s) reported:
$ merlin stop-workers
            *
       *~~~~~
     *~~*~~~*           __ __                  _ _
   /     ~~~~~         | \/ |                 | (_)
         ~~~~~         | \ / | ___ _ __| |_ _ __
        ~~~~~*         | |\/| |/ _ \ '__| | | '_ \
       *~~~~~~~        | | | | __/ | | | | | | |
     ~~~~~~~~~~        |_| |_|\___|_| |_|_|_| |_|
   *~~~~~~~~~~~
       ~~~*~~~*        Machine Learning for HPC Workflows
                                                                                                        (continues on next page)
28                                                                                       Chapter 1. Merlin Overview
                                                                              Merlin Documentation, Release 1.11.0
                                                                                                  (continued from previous page)
[2020-03-06 09:20:08: INFO] Stopping workers...
[2020-03-06 09:20:08: INFO] Reading app config from file .merlin/app.yaml
[2020-03-06 09:20:08: INFO] broker: amqps://user:******@server:5671//user
[2020-03-06 09:20:08: INFO] backend: redis://mlsi:******@server:6379/0
all_workers: ['celery@default_worker.%machine']
spec_worker_names: []
workers_to_stop: ['celery@default_worker.%machine']
[2020-03-06 09:20:10: INFO] Sending stop to these workers: ['celery@default_worker.
 ˓→%machine']
Using Samples
It’s a little boring to say “hello world” in just two different ways. Let’s instead say hello to many people!
To do this, we’ll need samples. Specifically, we’ll change WORLD from a global parameter to a sample. While parameters
are static, samples are generated dynamically, and can be more complex data types. In this case, WORLD will go from
being “world” or “mundo” to being a randomly-generated name.
First, we remove the global parameter WORLD so it does not conflict with our new sample. Parameters now look like
this:
global.parameters:
       GREET:
              values : ["hello", "hola"]
              label : GREET.%%
Now add these yaml sections to your spec:
env:
       variables:
              N_SAMPLES: 3
This makes N_SAMPLES into a user-defined variable that you can use elsewhere in your spec.
merlin:
       samples:
              generate:
                   cmd: python3 $(SPECROOT)/make_samples.py --filepath=$(MERLIN_INFO)/samples.
 ˓→csv --number=$(N_SAMPLES)
              file: $(MERLIN_INFO)/samples.csv
              column_labels: [WORLD]
This is the merlin block, an exclusively merlin feature. It provides a way to generate samples for your workflow. In this
case, a sample is the name of a person.
For simplicity we give column_labels the name WORLD, just like before.
It’s also important to note that $(SPECROOT) and $(MERLIN_INFO) are reserved variables. The $(SPECROOT) variable
is a shorthand for the directory path of the spec file and the $(MERLIN_INFO) variable is a shorthand for the directory
holding the provenance specs and sample generation results. More information on Merlin variables can be found on
the variables page.
1.1. Tutorial                                                                                                                29
Merlin Documentation, Release 1.11.0
It’s good practice to shift larger chunks of code to external scripts. At the same location of your spec, make a new file
called make_samples.py:
import argparse
import names
import numpy as np
# argument parsing
parser = argparse.ArgumentParser(description="Make some samples (names of people).")
parser.add_argument("--number", type=int, action="store", help="the number of samples␣
 ˓→you want to make")
parser.add_argument("--filepath", type=str, help="output file")
args = parser.parse_args()
# sample making
all_names = np.loadtxt(names.FILES["first:female"], dtype=str, usecols=0)
selected_names = np.random.choice(all_names, size=args.number)
result = ""
name_list = list(selected_names)
result = "\n".join(name_list)
with open(args.filepath, "w") as f:
      f.write(result)
Since our environment variable N_SAMPLES is set to 3, this sample-generating command should churn out 3 different
names.
Before we can run this, we must install the script’s external python library dependencies (names: a simple package that
generates random names, and numpy: a scientific computing package):
$ pip3 install -r requirements.txt
Here’s our DAG with samples:
30                                                                                      Chapter 1. Merlin Overview
                                                                 Merlin Documentation, Release 1.11.0
Here’s your new and improved my_hello.yaml, which now should match hello_samples.yaml:
description:
     name: hello_samples
     description: a very simple merlin workflow, with samples
env:
     variables:
          N_SAMPLES: 3
global.parameters:
     GREET:
          values : ["hello","hola"]
          label : GREET.%%
study:
     - name: step_1
        description: say hello
        run:
             cmd: echo "$(GREET), $(WORLD)!"
     - name: step_2
        description: print a success message
        run:
             cmd: print("Hurrah, we did it!")
             depends: [step_1_*]
             shell: /usr/bin/env python3
merlin:
     samples:
          generate:
                cmd: python3 $(SPECROOT)/make_samples.py --filepath=$(MERLIN_INFO)/samples.
 ˓→csv --number=$(N_SAMPLES)
          file: $(MERLIN_INFO)/samples.csv
          column_labels: [WORLD]
Run the workflow again!
Once finished, this is what the insides of step_1 look like:
1.1. Tutorial                                                                                      31
Merlin Documentation, Release 1.11.0
    • Numerically-named directories like 00, 01, and 02 are sample directories. Instead of storing sample output in
       a single flattened location, merlin stores them in a tree-like sample index, which helps get around file system
       constraints when working with massive amounts of data.
Lastly, let’s flex merlin’s muscle a bit and scale up our workflow to 1000 samples. To do this, you could internally
change the value in the spec from 3 to 1000. OR you could just run this:
$ merlin run my_hello.yaml --vars N_SAMPLES=1000
$ merlin run-workers my_hello.yaml
Once again, to send a warm stop signal to your workers, run:
$ merlin stop-workers
Congratulations! You concurrently greeted 1000 friends in English and Spanish!
1.1.5 Run a Real Simulation
Summary
This module aims to do a parameter study on a well-known benchmark problem for viscous incompressible fluid flow.
Prerequisites
    • Module 0: Before you start
    • Module 2: Installation
    • Module 3: Hello World
Estimated time
    • 60 minutes
You will learn
    • How to run the simulation OpenFOAM, using merlin.
    • How to use machine learning on OpenFOAM results, using merlin.
 Table of Contents:
      • Introduction
            – Before Moving On
      • Specification file
            – Variables
32                                                                                   Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
            – Samples and scripts
            – Setting up
            – Running the simulation
            – Combining outputs
            – Machine Learning and visualization
            – Putting it all together
      • Run the workflow
Introduction
We aim to do a parameter study on the lid-driven cavity problem.
       Fig. 1: Fig 1. Lid-driven cavity problem setup             Fig. 2: Fig 2. Example of a flow in steady state
In this problem, we have a viscous fluid within a square cavity that has three non-slip walls and one moving wall
(moving lid). We are interested in learning how varying the viscosity and lid speed affects the average enstrophy and
kinetic energy of the fluid after it reaches steady state. We will be using the velocity squared as a proxy for kinetic
energy.
This module will be going over:
    • Setting up our inputs using the merlin block
    • Running multiple simulations in parallel
    • Combining the outputs of these simulations into a an array
    • Predictive modeling and visualization
1.1. Tutorial                                                                                                       33
Merlin Documentation, Release 1.11.0
Before Moving On
check that the virtual environment with merlin installed is activated and that redis server is set up using this command:
$ merlin info
This is covered more in depth here: Checking/Verifying Installation
There are two ways to do this example: with docker and without docker. To go through the version with docker, get the
necessary files for this module by running:
$ merlin example openfoam_wf
$ cd openfoam_wf/
For the version without docker you should run:
$ merlin example openfoam_wf_no_docker
$ cd openfoam_wf_no_docker/
Note: From here on, this tutorial will focus solely on the docker version of running openfoam. However, the docker
version of this tutorial is almost identical to the no docker version. If you’re using the no docker version of this tutorial
you can still follow along but check the openfoam_no_docker_template.yaml file in each step to see what differs.
In the openfoam_wf directory you should see the following:
     • openfoam_wf.yaml – this spec file is partially blank. You will fill in the gaps as you follow this module’s steps.
     • openfoam_wf_template.yaml – this is a complete spec file. You can always reference it as an example.
     • scripts – This directory contains all the necessary scripts for this module.
          – We’ll be exploring these scripts as we go with the tutorial.
     • requirements.txt – this is a text file listing this workflow’s python dependencies.
To start, open openfoam_wf.yaml using your favorite text editor.
It should look something like this:
                                                Listing 1: openfoam_wf.yaml
description:
      name: openfoam_wf
      description: |
        A parameter study that includes initializing, running,
        post-processing, collecting, learning and visualizing OpenFOAM runs
        using docker.
env:
      variables:
           OUTPUT_PATH:
           SCRIPTS:
                                                                                                       (continues on next page)
34                                                                                        Chapter 1. Merlin Overview
                                                  Merlin Documentation, Release 1.11.0
              Fig. 3: Fig 3. openfoam_wf directory structure
1.1. Tutorial                                                                       35
Merlin Documentation, Release 1.11.0
                                                                        (continued from previous page)
         N_SAMPLES:
merlin:
     samples:
         generate:
             cmd: |
         file:
         column_labels:
     resources:
         workers:
             nonsimworkers:
                 args: -l INFO --concurrency <INPUT CONCURRENCY HERE>
                 steps:
             simworkers:
                 args: -l INFO --concurrency <INPUT CONCURRENCY HERE> --prefetch-
 ˓→multiplier 1 -Ofair
                 steps:
study:
     - name: setup
       description: |
                 Installs necessary python packages and imports the cavity directory
                 from the docker container
       run:
           cmd: |
     - name: sim_runs
       description: |
                    Edits the Lidspeed and viscosity then runs OpenFOAM simulation
                    using the icoFoam solver
       run:
           cmd: |
           depends:
           task_queue: simqueue
     - name: combine_outputs
       description: |
                 Combines the outputs of the previous step
       run:
           cmd: |
           depends:
     - name: learn
       description: |
                 Learns the output of the openfoam simulations using input parameters
                 and outputs error visualization from the experiment
       run:
           cmd: |
                                                                               (continues on next page)
36                                                                Chapter 1. Merlin Overview
                                                                         Merlin Documentation, Release 1.11.0
                                                                                             (continued from previous page)
               depends:
Specification file
We are going to build a spec file that produces this DAG:
                                              Fig. 4: Fig 4. Module 4 DAG
Variables
First we specify some variables to make our life easier. Locate the env block in our yaml spec
env:
      variables:
            OUTPUT_PATH: ./openfoam_wf_output
            SCRIPTS:
            N_SAMPLES:
The OUTPUT_PATH variable is set to tell merlin where you want your output directory to be.                The default is
<spec_name>_<TIMESTAMP> which in our case would simply be openfoam_wf_<TIMESTAMP>
We’ll fill out the next two variables as we go.
1.1. Tutorial                                                                                                           37
Merlin Documentation, Release 1.11.0
Samples and scripts
One merlin best practice is to copy any scripts your workflow may use from your SPECROOT directory into the
MERLIN_INFO directory. This is done to preserve the original scripts in case they are modified during the time merlin
is running. We will do that first. We will put this in the merlin sample generation section, since it runs before anything
else.
Edit the merlin block to look like the following:
merlin:
      samples:
            generate:
                  cmd: |
                       cp -r $(SPECROOT)/scripts $(MERLIN_INFO)/
                       # Generates the samples
                       python $(MERLIN_INFO)/scripts/make_samples.py -n 10 -outfile=$(MERLIN_
 ˓→INFO)/samples
            file: $(MERLIN_INFO)/samples.npy
            column_labels: [LID_SPEED, VISCOSITY]
We will be using the scripts directory a lot so we’ll set the variable SCRIPTS to $(MERLIN_INFO)/scripts for con-
venience. We would also like to have a more central control over the number of samples generated so we’ll create an
N_SAMPLES variable:
env:
      variables:
            OUTPUT_PATH: ./openfoam_wf_output
            SCRIPTS: $(MERLIN_INFO)/scripts
            N_SAMPLES: 10
and update the merlin block to be:
merlin:
      samples:
            generate:
                  cmd: |
                       cp -r $(SPECROOT)/scripts $(MERLIN_INFO)/
                       # Generates the samples
                       python $(SCRIPTS)/make_samples.py -n N_SAMPLES -outfile=$(MERLIN_INFO)/
 ˓→samples
            file: $(MERLIN_INFO)/samples.npy
            column_labels: [LID_SPEED, VISCOSITY]
Just like in the Using Samples step of the hello world module, we generate samples using the merlin block. We are only
concerned with how the variation of two initial conditions, lidspeed and viscosity, affects outputs of the system. These
are the column_labels. The make_samples.py script is designed to make log uniform random samples. Now, we
can move on to the steps of our study block.
38                                                                                     Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
Setting up
Our first step in our study block is concerned with making sure we have all the required python packages for this
workflow. The specific packages are found in the requirements.txt file.
We will also need to copy the lid driven cavity deck from the OpenFOAM docker container and adjust the write controls.
This last part is scripted already for convenience.
Locate the setup step in the study block and edit it to look like the following:
study:
  - name: setup
     description: |
                    Installs necessary python packages and imports the cavity directory
                    from the docker container
     run:
        cmd: |
           pip install -r $(SPECROOT)/requirements.txt
           # Set up the cavity directory in the MERLIN_INFO directory
           source $(SCRIPTS)/cavity_setup.sh $(MERLIN_INFO)
This step does not need to be parallelized so we will assign it to lower concurrency (a setting that controls how many
workers can be running at the same time)
Locate the resources section in the merlin block and edit the concurrency and add the setup step:
resources:
     workers:
           nonsimworkers:
                  args: -l INFO --concurrency 1
                  steps: [setup]
Running the simulation
Moving on to the sim_runs step, we want to:
   1. Copy the cavity deck from the MERLIN_INFO directory into each of the current step’s subdirectories
   2. Edit the default input values (lidspeed and viscosity) in these cavity decks using the sed command
   3. Run the simulation using the run_openfoam executable through the OpenFOAM docker container
   4. Post-process the results (also using the run_openfoam executable)
This part should look like:
- name: sim_runs
  description: |
                    Edits the Lidspeed and viscosity then runs OpenFOAM simulation
                    using the icoFoam solver
  run:
        cmd: |
              cp -r $(MERLIN_INFO)/cavity cavity/
              cd cavity
                                                                                                    (continues on next page)
1.1. Tutorial                                                                                                            39
Merlin Documentation, Release 1.11.0
                                                                                                (continued from previous page)
                ## Edits default values for viscosity and              lidspeed with
                # values specified by samples section of               the merlin block
                sed -i '' "18s/.*/nu                        [0 2 -1    0 0 0 0] $(VISCOSITY);/" constant/
 ˓→transportProperties
                sed -i '' "26s/.*/                value                uniform ($(LID_SPEED) 0 0);/" 0/U
                cd ..
                cp $(SCRIPTS)/run_openfoam .
                # Creating a unique OpenFOAM docker container for each sample and using it to␣
 ˓→run the simulation
                CONTAINER_NAME='OPENFOAM_ICO_$(MERLIN_SAMPLE_ID)'
                docker container run -ti --rm -v $(pwd):/cavity -w /cavity --name=${CONTAINER_
 ˓→NAME} cfdengine/openfoam ./run_openfoam $(LID_SPEED)
                docker wait ${CONTAINER_NAME}
          depends: [setup]
          task_queue: simqueue
This step runs many simulations in parallel so it would run faster if we assign it a worker with a higher concurrency.
Navigate back to the resources section in the merlin block
resources:
       workers:
            nonsimworkers:
                  args: -l INFO --concurrency 1
                  steps: [setup]
            simworkers:
                  args: -l INFO --concurrency 10 --prefetch-multiplier 1 -Ofair
                  steps: [sim_runs]
The quantities of interest are the average enstrophy and kinetic energy at each cell. The enstrophy is calculated through
an OpenFOAM post processing function of the the flow fields while the kinetic energy is approximated by calculated
using the square of the velocity vector at each grid point. The velocity field is normally outputted normally as a result
of running the default solver for this particular problem.
The run_openfoam executable calculates the appropriate timestep deltaT so that we have a Courant number of less
than 1. It also uses the icoFoam solver on the cavity decks and gives us VTK files that are helpful for visualizing the
flow fields using visualization tools such as VisIt or ParaView.
Combining outputs
Navigate to the next step in our study block combine_outputs. The purpose of this step is to extracts the data from
each of the simulation runs from the previous step (sim_runs) and combines it for future use.
The combine_outputs.py script in the $(SCRIPTS) directory is provided for convenience. It takes two inputs. The
first informs it of the base directory of the sim_runs directory and the second specifies the subdirectories for each run.
The script then goes into each of the directories and combines the velocity and enstrophy for each timestep of each run
in a .npz file.
- name: combine_outputs
   description: Combines the outputs of the previous step
   run:
                                                                                                       (continues on next page)
40                                                                                      Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
                                                                                              (continued from previous page)
        cmd: |
              python $(SCRIPTS)/combine_outputs.py -data $(sim_runs.workspace) -merlin_paths
 ˓→$(MERLIN_PATHS_ALL)
        depends: [sim_runs_*]
This step depends on all the previous step’s simulation runs which is why we have the star. However, it does not need
to be parallelized so we assign it to the nonsimworkers in the workers section of the merlin block.
workers:
      nonsimworkers:
           args: -l INFO --concurrency 1
           steps: [setup, combine_outputs]
Machine Learning and visualization
In the learn step, we want to:
    1. Post-process the .npz file from the previous step.
    2. Learn the mapping between our inputs and chosen outputs
    3. Graph important features
The provided learn.py script does all the above and outputs the trained sklearn model and a png of the graphs plotted
in the current directory.
- name: learn
   description: Learns the output of the openfoam simulations using input parameters
   run:
        cmd: |
              python $(SCRIPTS)/learn.py -workspace $(MERLIN_WORKSPACE)
        depends: [combine_outputs]
This step is also dependent on the previous step for the .npz file and will only need one worker therefore we will:
nonsimworkers:
      args: -l INFO --concurrency 1
      steps: [setup, combine_outputs, learn]
Putting it all together
By the end, your openfoam_wf.yaml should look like the template version in the same directory:
                                         Listing 2: openfoam_wf_template.yaml
description:
      name: openfoam_wf_template
      description: |
        A parameter study that includes initializing, running,
        post-processing, collecting, learning and visualizing OpenFOAM runs
        using docker.
                                                                                                     (continues on next page)
1.1. Tutorial                                                                                                             41
Merlin Documentation, Release 1.11.0
                                                                       (continued from previous page)
env:
     variables:
         OUTPUT_PATH: ./openfoam_wf_output
         SCRIPTS: $(MERLIN_INFO)/scripts
         N_SAMPLES: 100
merlin:
     samples:
         generate:
             cmd: |
                 cp -r $(SPECROOT)/scripts $(MERLIN_INFO)/
                 # Generates the samples
                 python $(SCRIPTS)/make_samples.py -n $(N_SAMPLES) -outfile=$(MERLIN_
 ˓→INFO)/samples
         file: $(MERLIN_INFO)/samples.npy
         column_labels: [LID_SPEED, VISCOSITY]
     resources:
         workers:
             nonsimworkers:
                 args: -l INFO --concurrency 1
                 steps: [setup, combine_outputs, learn]
             simworkers:
                 args: -l INFO --concurrency 10 --prefetch-multiplier 1 -Ofair
                 steps: [sim_runs]
study:
     - name: setup
       description: |
                 Installs necessary python packages and imports the cavity directory
                 from the docker container
       run:
           cmd: |
               pip install -r $(SPECROOT)/requirements.txt
               # Set up the cavity directory in the MERLIN_INFO directory
               source $(SCRIPTS)/cavity_setup.sh $(MERLIN_INFO)
     - name: sim_runs
       description: |
                   Edits the Lidspeed and viscosity then runs OpenFOAM simulation
                   using the icoFoam solver
       run:
           cmd: |
               cp -r $(MERLIN_INFO)/cavity cavity/
               cd cavity
               ## Edits default values for viscosity and lidspeed with
                                                                              (continues on next page)
42                                                               Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
                                                                                            (continued from previous page)
                  # values specified by samples section of the merlin block
                  sed -i '' "18s/.*/nu                           [0 2 -1 0 0 0 0] $(VISCOSITY);/"␣
 ˓→constant/transportProperties
                  sed -i '' "26s/.*/                  value                 uniform ($(LID_SPEED) 0 0);/" 0/U
                  cd ..
                  cp $(SCRIPTS)/run_openfoam .
                  # Creating a unique OpenFOAM docker container for each sample and using it␣
 ˓→to run the simulation
                  CONTAINER_NAME='OPENFOAM_ICO_$(MERLIN_SAMPLE_ID)'
                  docker container run -ti --rm -v $(pwd):/cavity -w /cavity --name=$
 ˓→{CONTAINER_NAME} cfdengine/openfoam ./run_openfoam $(LID_SPEED)
                  docker wait ${CONTAINER_NAME}
             depends: [setup]
             task_queue: simqueue
      - name: combine_outputs
        description: Combines the outputs of the previous step
        run:
             cmd: |
                  python $(SCRIPTS)/combine_outputs.py -data $(sim_runs.workspace) -merlin_
 ˓→paths $(MERLIN_PATHS_ALL)
             depends: [sim_runs_*]
      - name: learn
        description: Learns the output of the openfoam simulations using input parameters
        run:
             cmd: |
                  python $(SCRIPTS)/learn.py -workspace $(MERLIN_WORKSPACE)
             depends: [combine_outputs]
Run the workflow
Now that you are done with the Specification file, use the following commands from inside the openfoam_wf directory
to run the workflow on our task server.
Note: Running with fewer samples is the one of the best ways to debug
$ merlin run openfoam_wf.yaml
$ merlin run-workers openfoam_wf.yaml
But wait! We realize that 10 samples is not enough to train a good model. We would like to restart with 100 samples
instead of 10 (should take about 6 minutes):
After sending the workers to start on their queues, we need to first stop the workers:
$ merlin stop-workers --spec openfoam_wf.yaml
1.1. Tutorial                                                                                                          43
Merlin Documentation, Release 1.11.0
Note:
    • The –spec flag only stops workers from a specific YAML spec
We stopped these tasks from running but if we were to run the workflow again (with 100 samples instead of 10), we
would continue running the 10 samples first! This is because the queues are still filled with the previous attempt’s tasks.
We need to purge these queues first in order to repopulate them with the appropriate tasks. This is where we use the
merlin purge command:
$ merlin purge openfoam_wf.yaml
Now we are free to repopulate the queues with the 100 samples:
$ merlin run openfoam_wf.yaml --vars N_SAMPLES=100
$ merlin run-workers openfoam_wf.yaml
To see your results, look inside the learn output directory. You should see a png that looks like this:
44                                                                                       Chapter 1. Merlin Overview
                                                                  Merlin Documentation, Release 1.11.0
Related articles
    • https://cfd.direct/openfoam/user-guide/v6-cavity/
    • https://www.cfdengine.com/blog/how-to-install-openfoam-anywhere-with-docker/
1.1. Tutorial                                                                                       45
Merlin Documentation, Release 1.11.0
1.1.6 Advanced Topics
Prerequisites
    • Module 2: Installation
    • Module 3: Hello World
    • Module 4: Running a Real Simulation
    • Python virtual environment containing the following packages
          – merlin
          – pandas
          – faker
          – maestrowf
Estimated time
    • 15 minutes
You will learn
    • Run workflows using HPC batch schedulers
    • Distribute workflows across multiple batch allocations and machines
    • Setup iterative workflow specs suited for optimization and dynamic sampling applications
 Table of Contents:
     • Setup
     • Interfacing with HPC systems
     • Multi-machine workflows
     • Dynamic task queuing and sampling
Setup
The code for the following examples can be obtained from command line, invoking:
merlin example hpc_demo
This will copy the three merlin workflow specifications from this section and the supporting python scripts. Each
specification may need some modification to adapt it to the batch scheduler you will be using. In addition, the dynamic
sampling workflow will need an additional modification to set the path of the virtual environment, which is set as a
variable in the env block.
46                                                                                    Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
Interfacing with HPC systems
Another block is added to the merlin workflow specification when running on HPC systems, the batch block. This
block contains information about the batch scheduler system such as batch type, batch queue to use, and banks to charge
to. There are additional optional arguments for addressing any special configurations or launch command arguments,
varying based on batch type. In addition, the shell type used by each steps cmd scripts can be specified here. The
number of nodes in a batch allocation can be defined here, but it will be overridden in the worker config.
batch:
    # Required keys:
    type: flux
    bank: testbank
    queue: pbatch
    # Optional keys:
    flux_path: <optional path to flux bin>
    flux_start_opts: <optional flux start options>
    flux_exec_workers: <optional, flux argument to launch workers on
                               all nodes. (True)>
    launch_pre: <Any configuration needed before the srun or jsrun launch>
    launch_args: <Optional extra arguments for the parallel launch command>
    worker_launch: <Override the parallel launch defined in merlin>
    shell: <the interpreter to use for the script after the shebang>
              # e.g. /bin/bash, /bin/tcsh, python, /usr/bin/env perl, etc.
    nodes: <num nodes> # The number of nodes to use for all workers
              This can be overridden in the workers config.
              If this is unset the number of nodes will be
              queried from the environment, failing that, the
              number of nodes will be set to 1.
Inside the study step specifications are a few additional keys that become more useful on HPC systems: nodes, procs,
and task_queue. Adding on the actual study steps to the above batch block specifies the actual resources each steps
processes will take.
study:
    - name: sim-runs
       description: Run simulations
       run:
           cmd: $(LAUNCHER) echo "$(VAR1) $(VAR2)" > simrun.out
           nodes: 4
           procs: 144
           task_queue: sim_queue
    - name: post-process
       description: Post-Process simulations on second allocation
       run:
           cmd: |
              cd $(runs1.workspace)/$(MERLIN_SAMPLE_PATH)
              $(LAUNCHER) <parallel-post-proc-script>
           nodes: 1
           procs: 36
                                                                                                    (continues on next page)
1.1. Tutorial                                                                                                            47
Merlin Documentation, Release 1.11.0
                                                                                              (continued from previous page)
           depends: [sim-runs]
           task_queue: post_proc_queue
In addition to the batch block is the resources section inside the merlin block. This can be used to put together
custom celery workers. Here you can override batch types and node counts on a per worker basis to accommodate
steps with different resource requirements. In addition, this is where the task_queue becomes useful, as it groups the
different allocation types, which can be assigned to each worker here by specifying step names.
Arguments to celery itself can also be defined here with the args key. Of particular interest will be:
                                     --concurrency                  <num_threads>
                                     --prefetch-multiplier          <num_tasks>
                                     -0 fair
Concurrency can be used to run multiple workers in an allocation, thus is recommended to be set to the number of
simulations or step work items that fit into the number of nodes in the batch allocation in which these workers are
spawned. Note that some schedulers, such as flux, can support more jobs than the node has resources for. This may
not impact the throughput, but it can prevent over-subscription errors that might otherwise stop the workflow.
The prefetch multiplier is more related to packing in tasks into the time of the allocation. For long running tasks it is
recommended to set this to 1. For short running tasks, this can reduce overhead from talking to the rabbit servers by
requesting <num_threads> x <num_tasks> tasks at a time from the server.
The -0 fair option enables workers running tasks from different queues to run on the same allocation.
The example block below extends the previous with workers configured for long running simulation jobs as well as
shorter running post processing tasks that can cohabit an allocation
merlin:
   resources:
      task_server: celery
      overlap: False
      # Customize workers
      workers:
           simworkers:
                 args: --concurrency 1
                 steps: [sim-runs]
                 nodes: 4
                 machines: [host1]
           postworkers:
                 args: --concurrency 4 --prefetch-multiplier 2
                 steps: [post-proc-runs]
                 nodes: 1
                 machines: [host1]
Putting it all together with the parameter blocks we have an HPC batch enabled study specification. In this demo
workflow, sample_names generates one many single core jobs, with concurrency set to 36 for this particular machine
that has 36 cores per node. The collect step on the other hand consists of a single job that uses all cores on the node,
and is assigned to a queue that has a concurrency of 1.
48                                                                                     Chapter 1. Merlin Overview
                                                        Merlin Documentation, Release 1.11.0
description:
   name: hpc_demo
   description: Demo running a workflow on HPC machines
env:
   variables:
     OUTPUT_PATH: ./name_studies
     # Collect individual sample files into one for further processing
     COLLECT: $(SPECROOT)/sample_collector.py
     # Process single iterations' results
     POST_PROC: $(SPECROOT)/sample_processor.py
     # Process all iterations
     CUM_POST_PROC: $(SPECROOT)/cumulative_sample_processor.py
     # Number of threads for post proc scripts
     POST_NPROCS: 36
     PYTHON: <INSERT PATH TO VIRTUALENV HERE>
batch:
    type: flux
    bank: testbank
    queue: pdebug
    shell: /bin/bash
    nodes: 1
########################################
# Study definition
########################################
study:
    - name: sample_names
      description: Record samples from the random name generator
      run:
         cmd: |
           $(LAUNCHER) echo "$(NAME)"
           $(LAUNCHER) echo "$(NAME)" > name_sample.out
         nodes: 1
         procs: 1
         task_queue: name_queue
    - name: collect
      description: Collect all samples generated
      run:
         cmd: |
           echo $(MERLIN_GLOB_PATH)
           echo $(sample_names.workspace)
           ls $(sample_names.workspace)/$(MERLIN_GLOB_PATH)/name_sample.out | xargs
 ˓→$(PYTHON) $(COLLECT) -out collected_samples.txt --np $(POST_NPROCS)
         nodes: 1
                                                                            (continues on next page)
1.1. Tutorial                                                                                    49
Merlin Documentation, Release 1.11.0
                                                                        (continued from previous page)
         procs: 1
         depends: [sample_names_*]
         task_queue: post_proc_queue
    - name: post-process
      description: Post-Process collection of samples, counting occurrences of unique␣
 ˓→names
      run:
         cmd: |
            $(PYTHON) $(POST_PROC) $(collect.workspace)/collected_samples.txt --results␣
 ˓→iter_$(ITER)_results.json
         nodes: 1
         procs: 1
         depends: [collect]
         task_queue: post_proc_queue
########################################
# Worker and sample configuration
########################################
merlin:
   resources:
     task_server: celery
     overlap: False
     workers:
         nameworkers:
              args: --concurrency 36 --prefetch-multiplier 3
              steps: [sample_names]
              nodes: 1
              machines: [borax, quartz]
         postworkers:
              args: --concurrency 1 --prefetch-multiplier 1
              steps: [post-process]
              nodes: 1
              machines: [borax, quartz]
   ###################################################
   samples:
     column_labels: [NAME]
     file: $(MERLIN_INFO)/samples.csv
     generate:
       cmd: |
         $(PYTHON) $(SPECROOT)/faker_sample.py -n 200 -outfile=$(MERLIN_INFO)/samples.csv
50                                                                Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
Multi-machine workflows
Spreading this workflow across multiple machines is a simple modification of the above workflow: simply add addi-
tional host names to the machines list in the worker config. A caveat for this feature is that all host systems will need to
have access to the same workspace/filesystem. The following resource block demonstrates usage of one host for larger
simulation steps, and a second host for the smaller post processing steps. In this case you simply need an alloc on each
host, and can simply execute run-workers on each, with run only needed once up front to send the tasks to the queue
server.
Dynamic task queuing and sampling
Iterative workflows, such as optimization or machine learning, can be implemented in merlin via recursive workflow
specifications that use dynamic task queuing. The example spec below is a simple implementation of this using an
iteration counter $(ITER) and a predetermined limit, $(MAX_ITER) to limit the number of times to generate new
samples and spawn a new instantiation of the workflow. The iteration counter takes advantage of the ability to override
workflow variables on the command line.
     description:
        name: dynamic_sampling_demo
        description: Demo dynamic sampling workflow
     env:
        variables:
          OUTPUT_PATH: ./name_studies
          ITER_OUTPUT: $(SPECROOT)/$(OUTPUT_PATH)/iter_outputs # Iteration and cumulative␣
 ˓→results
          COLLECT: $(SPECROOT)/sample_collector.py
          POST_PROC: $(SPECROOT)/sample_processor.py # Process single iterations' results
          CUM_POST_PROC: $(SPECROOT)/cumulative_sample_processor.py # Process all iterations
          POST_NPROCS: 36                   # Number of threads for post proc scripts
          PYTHON: /usr/WS2/white242/merlin_dev_2/venv_merlin_py3_7/bin/python
          ITER: 1
          MAX_ITER: 10
     batch:
         type: flux
         bank: testbank
         queue: pdebug
         shell: /bin/bash
         nodes: 1
     ########################################
     # Study definition
     ########################################
     study:
         - name: sample_names
           description: Record samples from the random name generator
           run:
               cmd: |
                  $(LAUNCHER) echo "$(NAME)"
                  $(LAUNCHER) echo "$(NAME)" > name_sample.out
               nodes: 1
               procs: 1
                                                                                                        (continues on next page)
1.1. Tutorial                                                                                                                51
Merlin Documentation, Release 1.11.0
                                                                       (continued from previous page)
            task_queue: name_queue
       - name: collect
         description: Collect all samples generated
         run:
            cmd: |
              echo $(MERLIN_GLOB_PATH)
              echo $(sample_names.workspace)
              ls $(sample_names.workspace)/$(MERLIN_GLOB_PATH)/name_sample.out | xargs
˓→$(PYTHON) $(COLLECT) -out collected_samples.txt --np $(POST_NPROCS)
            nodes: 1
            procs: 1
            depends: [sample_names_*]
            task_queue: post_proc_queue
       - name: post-process
         description: Post-Process collection of samples, counting occurrences of unique␣
˓→names
         run:
            cmd: |
              $(PYTHON) $(POST_PROC) $(collect.workspace)/collected_samples.txt --results
˓→$(ITER_OUTPUT)/iter_$(ITER)_results.json
            nodes: 1
            procs: 1
            depends: [collect]
            task_queue: post_proc_queue
       - name: run-more-samples
         description: Generate new set of samples and rerun, or generate some descriptive␣
˓→plots/statistics
         run:
           cmd: |
             if [ $(ITER) -ge $(MAX_ITER) ] ; then
                echo "done"
                $(PYTHON) $(CUM_POST_PROC) $(ITER_OUTPUT)/iter_*_results.json --np $(POST_
˓→NPROCS) --hardcopy $(ITER_OUTPUT)/cumulative_results.png
             else
                next_iter=$(ITER)
                ((next_iter=next_iter+1))
                echo "Starting iteration " $next_iter
                cd $(SPECROOT)
                merlin run $(SPECROOT)/faker_demo.yaml --vars ITER=$next_iter
             fi
           nodes: 1
           procs: 1
           depends: [post-process]
           task_queue: post_proc_queue
                                                                              (continues on next page)
52                                                               Chapter 1. Merlin Overview
                                                                             Merlin Documentation, Release 1.11.0
                                                                                                  (continued from previous page)
     ########################################
     # Worker and sample configuration
     ########################################
     merlin:
        resources:
          task_server: celery
          overlap: False
          # Customize workers NOTE: abuse this for scaling study: prefetch mult increase
          #     - celery->rabbit query overhead for fast jobs
          workers:
                nameworkers:
                     args: --concurrency 36 --prefetch-multiplier 3
                     steps: [sample_names]
                     nodes: 1
                     machines: [borax, quartz]
              # NOTE: specifying wrong step leaves orphaned queue -> purge first!
              # also, invalid host name appears to fail silently
                postworkers:
                     args: --concurrency 1 --prefetch-multiplier 1
                     steps: [post-process]
                     nodes: 1
                     machines: [borax, quartz]
        ###################################################
        samples:
          column_labels: [NAME]
          file: $(MERLIN_INFO)/samples.csv
          generate:
             cmd: |
                $(PYTHON) $(SPECROOT)/faker_sample.py -n 200 -outfile=$(MERLIN_INFO)/samples.
 ˓→csv
This workflow specification is intended to be invoke within an allocation of nodes on your HPC cluster, e.g. within
and sxterm. The last step to queue up new samples for the next iteration, merlin run faker_demo.yaml ..., only
doesn’t need to also call run-workers since the workers from the first instantiation are still alive. Thus the new samples
will immediately start processing on the existing allocation.
Another change in this workflow relative to the single stage version is managing the workspaces and outputs. The
strategy used here is to create a new directory for collecting each iterations final outputs, $(ITER_OUTPUT), facilitating
collective post processing at the end without having to worry about traversing into each iterations’ local workspaces.
The workflow itself isn’t doing anything practical; it’s simply repeatedly sampling from a fake name generator in an
attempt to count the number of unique names that are possible. The figure below shows results from running 20
iterations, with the number of unique names faker can generate appearing to be slightly more than 300.
1.1. Tutorial                                                                                                                53
Merlin Documentation, Release 1.11.0
1.1.7 Contribute to Merlin
Estimated time
    • 10 minutes
You will learn
    • How to post issues to the merlin repo.
    • Guidelines for contributing to merlin.
 Table of Contents:
     • Issues
           – Bug Reports
           – Feature Requests
           – Questions
54                                           Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
      • Contributing
Issues
Found a bug? Have an idea? Or just want to ask a question? Create a new issue on GitHub.
Bug Reports
To report a bug, simply navigate to Issues, click “New Issue”, then click “Bug report”. Then simply fill out a few fields
such as “Describe the bug” and “Expected behavior”. Try to fill out every field as it will help us figure out your bug
sooner.
Feature Requests
We are still adding new features to merlin. To suggest one, simply navigate to Issues, click “New Issue”, then click
“Feature request”. Then fill out a few fields such as “What problem is this feature looking to solve?”
Questions
Note: Who knows? Your question may already be answered in the FAQ.
We encourage questions to be asked in a collaborative setting: on GitHub, direct any questions to General Questions
in Issues.
Any questions can also be sent to merlin@llnl.gov.
Contributing
Merlin is an open source project, so contributions are welcome. Contributions can be anything from bugfixes, docu-
mentation, or even new core features.
Contributing to Merlin is easy! Just send us a pull request from your fork. Before you send it, summarize your change
in the [Unreleased] section of CHANGELOG.md and make sure develop is the destination branch. We also appreciate
squash commits before pull requests are merged.
Merlin uses a rough approximation of the Git Flow branching model. The develop branch contains the latest contribu-
tions, and main is always tagged and points to the latest stable release.
If you’re a contributor, try to test and run on develop. That’s where all the magic is happening (and where we hope
bugs stop).
More detailed information on contributing can be found on the Contributing page.
1.1. Tutorial                                                                                                         55
Merlin Documentation, Release 1.11.0
1.1.8 Port Your Own Application
Prerequisites
     • Module 2: Installation
     • Module 3: Hello World
     • Module 4: Running a Real Simulation
Estimated time
     • 15 minutes
You will learn
     • Tips for building workflows
     • Tips for scaling
     • Debugging
  Table of Contents:
       • Tips for porting your app, building workflows
       • Tips for debugging your workflows
       • Tips for scaling workflows
       • Misc tips
Tips for porting your app, building workflows
The first step of building a new workflow, or porting an existing app to a workflow, is to describe it as a set of discrete,
and ideally focused steps. Decoupling the steps and making them generic when possible will facilitate more rapid
composition of future workflows. This will also require mapping out the dependencies and parameters that get passed
between/shared across these steps.
Setting up a template using tools such as cookiecutter can be useful for more production style workflows that will
be frequently reused. Additionally, make use of the built-in examples accessible from the merlin command line with
merlin example.
Use dry runs merlin run --dry --local to prototype without actually populating task broker’s queues. Similarly,
once the dry run prototype looks good, try it on a small number of parameters before throwing millions at it.
Merlin inherits much of the input language and workflow specification philosophy from Maestro. Thus a good first step
is to learn to use that tool. As seen in the Module 5: Advanced Topics there are also use cases that combine Merlin and
Maestro.
Make use of exit keys such as MERLIN_RESTART or MERLIN_RETRY in your step logic.
56                                                                                     Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
Tips for debugging your workflows
The scripts defined in the workflow steps are also written to the output directories; this is a useful debugging tool as it
can both catch parameter and variable replacement errors, as well as provide a quick way to reproduce, edit, and retry
the step offline before fixing the step in the workflow specification. The <stepname>.out and <stepname>.err files
log all of the output to catch any runtime errors. Additionally, you may need to grep for 'WARNING' and 'ERROR' in
the worker logs.
When a bug crops up in a running study with many parameters, there are a few other commands to make use of. Rather
than trying to spam Ctrl-c to kill all the workers, you will want to instead use merlin stop-workers --spec
<workflow_name>.yaml to stop the workers for that workflow. This should then be followed up with merlin purge
<workflow_name>.yaml to clear out the task queue to prevent the same buggy tasks from continuing to run the next
time run-workers is invoked.
Tips for scaling workflows
Most of the worst bottlenecks you’ll encounter when scaling up your workflow are caused by the file system. This can
be caused by using too much space or too many files, even in a single workflow if you’re not careful. There is a certain
number of inodes created just based upon the sample counts even without accounting for the steps being executed. This
can be mitigated by avoiding reading/writing to the file system when possible. If file creation is unavoidable, you may
need to consider adding cleanup steps to your workflow: dynamically pack up the previous step in a tarball, transfer to
another file system or archival system, or even just delete files.
Misc tips
Avoid reliance upon storage at the $(SPECROOT) level. This is particularly dangerous if using symlinks as it can violate
the provenance of what was run, possibly ruining the utility of the dataset that was generated. It is preferred to make
local copies of any input decks and supporting scripts and data sets inside the workflows’ workspace. This of course has
limits, regarding shared/system libraries that any programs running in the steps may need; alternate means of recording
this information in a log file or something similar may be needed in this case.
1.2 Getting Started
1.2.1 Quick Start
pip3 install merlin
All set up? See the Merlin Commands section for using merlin.
Check out the Tutorial!
1.2.2 Developer Setup
The developer setup can be done via pip or via make. This section will cover how to do both.
Additionally, there is an alternative method to setup merlin on supercomputers. See the Spack section for more details.
1.2. Getting Started                                                                                                   57
Merlin Documentation, Release 1.11.0
Pip Setup
To install with the additional developer dependencies, use:
pip3 install "merlin[dev]"
or:
pip3 install -e "git+https://github.com/LLNL/merlin.git@develop#egg=merlin[dev]"
Make Setup
Visit the Merlin repository on github. Create a fork of the repo and clone it onto your system.
Change directories into the merlin repo:
$ cd merlin/
Install Merlin with the developer dependencies:
$ make install-dev
This will create a virtualenv, start it, and install Merlin and it’s dependencies for you.
More documentation about using Virtualenvs with Merlin can be found at Using Virtualenvs with Merlin.
We can make sure it’s installed by running:
$ merlin --version
If you don’t see a version number, you may need to restart your virtualenv and try again.
Configuring Merlin
Once Merlin has been installed, the installation needs to be configured. Documentation for merlin configuration is in
the Configuring Merlin section.
That’s it. To start running Merlin see the Merlin Workflows.
(Optional) Testing Merlin
  Warning: With python 3.6 you may see some tests fail and a unicode error presented. To fix this, you need to
  reset the LC_ALL environment variable to en_US.utf8.
If you have make installed and the Merlin repository cloned, you can run the test suite provided in the Makefile by
running:
$ make tests
This will run both the unit tests suite and the end-to-end tests suite.
If you’d just like to run the unit tests you can run:
58                                                                                        Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
$ make unit-tests
Similarly, if you’d just like to run the end-to-end tests you can run:
$ make e2e-tests
1.2.3 Custom Setup
This section documents how to install Merlin without using the Makefile. This setup is more complicated; however,
allows for more customization of the setup configurations.
Clone the Merlin repository:
git clone https://github.com/LLNL/merlin.git
Create a virtualenv
Merlin uses virtualenvs to manage package dependencies which can be installed via Pip, Python’s default package
manager.
More documentation about using Virtualenvs with Merlin can be found at Using Virtualenvs with Merlin.
To create a new virtualenv and activate it:
$ python3 -m venv venv_merlin_$SYS_TYPE_py3_6
$ source venv_merlin_$SYS_TYPE_py3/bin/activate                     # Or activate.csh for .cshrc
Install Python Package Dependencies
Merlin uses Pip to manage Python dependencies. Merlin dependencies can be found in the requirements directory in
the Merlin repository.
To install the standard set of dependencies run:
(merlin3_7) $ pip install -r requirements.txt
This will install all the required dependencies for Merlin and development development dependencies.
Installing Merlin
Merlin can be installed in editable mode. From within the Merlin repository:
(merlin3_7) $ pip install -e .
Any changes made to the Merlin source code should automatically reflect in the virtualenv.
Tip: If changes to Merlin’s source code do not reflect when running Merlin try running pip install -e . from within
the Merlin repository.
1.2. Getting Started                                                                                            59
Merlin Documentation, Release 1.11.0
1.3 FAQ
 Frequently Asked Questions
    • General
         – What is Merlin?
         – Where can I get help with Merlin?
    • Setup & Installation
         – How can I build Merlin?
         – Do I have to build Merlin?
         – What are the setup instructions at LLNL?
         – How do I reconfigure for different servers?
    • Component Technology
         – What underlying libraries does Merlin use?
         – What security features are in Merlin?
         – What is celery?
         – What is maestro?
    • Designing and Building Workflows
         – Where are some example workflows?
         – How do I launch a workflow?
         – How do I describe workflows in Merlin?
         – What is a DAG?
         – What if my workflow can’t be described by a DAG?
         – How do I implement workflow looping / iteration?
         – Can steps be restarted?
         – How do I put a time delay in before a restart or retry?
         – I have a long running batch task that needs to restart, what should I do?
         – How do I mark a step failure?
         – What fields can be added to steps?
         – How do I specify the language used in a step?
    • Running Workflows
         – How do I set up a workspace without executing step scripts?
         – How do I start workers?
         – How do I see what workers are connected?
         – How do I stop workers?
         – How do I re-run failed steps in a workflow?
60                                                                                 Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
           – What tasks are in my queue?
           – How do I purge tasks?
           – Why is stuff still running after I purge?
           – Why am I running old tasks?
           – Where do tasks get run?
           – Can I run different steps from my workflow on different machines?
           – What is Slurm?
           – What is LSF?
           – What is flux?
           – What is PBS?
           – How do I use flux on LC?
           – What is LAUNCHER?
           – How do I use LAUNCHER?
           – What is level_max_dirs?
           – What is pgen?
           – Where can I learn more about merlin?
1.3.1 General
What is Merlin?
Merlin is a distributed task queue system designed to facilitate the large scale execution of HPC ensembles, like those
needed to build machine learning models of complex simulations.
Where can I get help with Merlin?
In addition to this documentation, the Merlin developers can be reached at merlin@llnl.gov. You can also reach out to
the merlin user group mailing list: merlin-users@listserv.llnl.gov.
1.3.2 Setup & Installation
How can I build Merlin?
Merlin can be installed via pip in a python virtual environment or via spack.
See Getting started.
1.3. FAQ                                                                                                            61
Merlin Documentation, Release 1.11.0
Do I have to build Merlin?
If you’re at LLNL and want to run on LC, you can use one of the public deployments. For more information, check out
the LLNL access page in confluence.
What are the setup instructions at LLNL?
See “Do I have to build Merlin”
How do I reconfigure for different servers?
The server configuration is set in ~/.merlin/app.yaml. Details can be found here.
1.3.3 Component Technology
What underlying libraries does Merlin use?
     • Celery
               – What is celery?
     • Maestro
               – What is maestro?
What security features are in Merlin?
Merlin encrypts network traffic of step results, implying that all results are encrypted with a unique user-based key,
which is auto-generated and placed in ~/.merlin/. This allows for multiple users to share a results database. This is
important since some backends, like redis do not allow for multiple distinct users.
What is celery?
Celery is an asynchronous task/job queue based on distributed message passing. It is focused on real-time operation,
but supports scheduling as well. See Celery’s GitHub page and Celery’s website for more details.
What is maestro?
Maestro is a tool and library for specifying and conducting general workflows. See Maestro’s GitHub page for more
details.
62                                                                                    Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
1.3.4 Designing and Building Workflows
yaml specification file
Where are some example workflows?
$ merlin example list
How do I launch a workflow?
To launch a workflow locally, use merlin run --local <spec>. To launch a distributed workflow, use merlin
run-workers <spec>, and merlin run <spec>. These may be done in any order.
How do I describe workflows in Merlin?
A Merlin workflow is described with a yaml specification file.
What is a DAG?
DAG is an acronym for ‘directed acyclic graph’. This is the way your workflow steps are represented as tasks.
What if my workflow can’t be described by a DAG?
There are certain workflows that cannot be explicitly defined by a single DAG; however, in our experience, many can.
Furthermore, those workflows that cannot usually do employ DAG sub-components. You probably can gain much of
the functionality you want by combining a DAG with control logic return features (like step restart and additional calls
to merlin run).
How do I implement workflow looping / iteration?
Combining exit $(MERLIN_RETRY) with max_retries can allow you to loop a single step. Entire workflow looping
/ iteration can be accomplished by finishing off your DAG with a final step that makes another call to merlin run.
Can steps be restarted?
Yes. To build this into a workflow, use exit $(MERLIN_RETRY) within a step to retry a failed cmd section. The max
number of retries in given step can be specified with the max_retries field.
Alternatively, use exit $(MERLIN_RESTART) to run the optional <step>.run.restart section.
To delay a retry or restart directive, add the retry_delay field to the step. Note: retry_delay only works in server
mode (ie not --local mode).
To restart failed steps after a workflow is done running, see How do I re-run failed steps in a workflow?.
1.3. FAQ                                                                                                             63
Merlin Documentation, Release 1.11.0
How do I put a time delay in before a restart or retry?
Add the retry_delay field to the step. This specifies how many seconds before the task gets run after the restart. Set
this value to large enough for your problem to finish.
See the merlin example restart_delay example for syntax.
Note: retry_delay only works in server mode (ie not --local mode).
I have a long running batch task that needs to restart, what should I do?
Before your allocation ends, use $(MERLIN_RESTART) or $(MERLIN_RETRY) but with a retry_delay on your step
for longer that your allocation has left. The server will hold onto the step for that long (in seconds) before releasing it,
allowing your batch allocation to end without the worker grabbing the step right away.
For instance, your step could look something like this
name: batch_task
description: A long running task that needs to restart
run:
      cmd: |
           # Run my code, but end 60 seconds before my allocation
           my_code --end_early 60s
           if [ -e restart_needed_flag ]; then
                 exit $(MERLIN_RESTART)
           fi
      retry_delay: 120 # wait at least 2 minutes before restarting
How do I mark a step failure?
Each step is ultimately designated as: * a success $(MERLIN_SUCCESS) – writes a MERLIN_FINISHED file to the
step’s workspace directory * a soft failure $(MERLIN_SOFT_FAIL) – allows the workflow to continue * a hard failure
$(MERLIN_HARD_FAIL) – stops the whole workflow by shutting down all workers on that step
Normally this happens behinds the scenes, so you don’t need to worry about it. To hard-code this into your step logic,
use a shell command such as exit $(MERLIN_HARD_FAIL).
Note: The $(MERLIN_HARD_FAIL) exit code will shutdown all workers connected to the queue associated with the
failed step. To shutdown all workers use the $(MERLIN_STOP_WORKERS) exit code
To rerun all failed steps in a workflow, see How do I re-run failed steps in a workflow?. If you really want a previously
successful step to be re-run, you can first manually remove the MERLIN_FINISHED file.
64                                                                                       Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
What fields can be added to steps?
Steps have a name, description, and run field, as shown below.
name: <string>
description: <string>
run:
     cmd: <shell command for this step>
Also under run, the following fields are optional:
run:
     depends: <list of step names>
     task_queue: <task queue name for this step>
     shell: <e.g., /bin/bash, /usr/bin/env python3>
     max_retries: <integer>
     retry_delay: <integer: seconds>
     nodes: <integer>
     procs: <integer>
How do I specify the language used in a step?
You can add the field shell under the run portion of your step to change the language you write your step in. The default
is /bin/bash, but you can do things like /usr/bin/env python as well. Use merlin example feature_demo to
see an example of this.
1.3.5 Running Workflows
$ merlin run <yaml file>
For more details, see Merlin commands.
How do I set up a workspace without executing step scripts?
$ merlin run --dry <yaml file>
How do I start workers?
$ merlin run-workers <yaml file>
1.3. FAQ                                                                                                              65
Merlin Documentation, Release 1.11.0
How do I see what workers are connected?
$ merlin query-workers
This command gives you fine control over which workers you’re looking for via a regex on their name, the queue names
associated with workers, or even by providing the name of a spec file where workers are defined.
For more info, see Searching for any workers (merlin query-workers).
How do I stop workers?
Interactively outside of a workflow (e.g. at the command line), you can do this with
$ merlin stop-workers
This gives you fine control over which kinds of workers to stop, for instance via a regex on their name, or the queue
names you’d like to stop.
From within a step, you can exit with the $(MERLIN_STOP_WORKERS) code, which will issue a time-delayed call to
stop all of the workers, or with the $(MERLIN_HARD_FAIL) directive, which will stop all workers connected to the
current step. This helps prevent the suicide race condition where a worker could kill itself before removing the step
from the workflow, causing the command to be left there for the next worker and creating a really bad loop.
You can of course call merlin stop-workers from within a step, but be careful to make sure the worker executing
it won’t be stopped too.
For more tricks, see Stopping workers (merlin stop-workers).
How do I re-run failed steps in a workflow?
$ merlin restart <spec>
What tasks are in my queue?
How do I purge tasks?
$ merlin purge <yaml file>
Why is stuff still running after I purge?
You probably have workers executing tasks. Purging removes them from the server queue, but any currently running
or reserved tasks are being held by the workers. You need to shut down these workers first:
$ merlin stop-workers
$ merlin purge <yaml file>
66                                                                                   Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
Why am I running old tasks?
You might have old tasks in your queues. Try merlin purge <yaml>. You might also have rogue workers. To find
out, try merlin query-workers.
Where do tasks get run?
Can I run different steps from my workflow on different machines?
Yes. Under the merlin block you can specify which machines your workers are allowed on. In order for this to work,
you must then use merlin run-workers separately on each of the specified machines.
merlin:
    resources:
         workers:
            worker_name:
                 machines: [hostA, hostB, hostC]
What is Slurm?
A job scheduler. See Slurm documentation .
What is LSF?
Another job scheduler. See IBM’s LSF documentation .
What is flux?
Flux is a hierarchical scheduler and launcher for parallel simulations. It allows the user to specify the same launch
command that will work on different HPC clusters with different default schedulers such as SLURM or LSF. Merlin
versions earlier than 1.9.2 used the non-flux native scheduler to launch a flux instance. Subsequent merlin versions can
launch the merlin workers using a native flux scheduler. More information can be found at the Flux web page.
Older versions of flux may need the --mpi=none argument if flux is launched on a system using the SLURM scheduler.
This argument can be added in the launch_args variable in the batch section.
batch:
   type: flux
   launch_args: --mpi=none
What is PBS?
Another job scheduler. See Portable Batch System . This functionality is only available to launch a flux scheduler.
1.3. FAQ                                                                                                              67
Merlin Documentation, Release 1.11.0
How do I use flux on LC?
The --mpibind=off option is currently required when using flux with a slurm launcher on LC toss3 systems. Set this
in the batch section as shown in the example below.
batch:
   type: flux
   launch_args: --mpibind=off
What is LAUNCHER?
$LAUNCHER is a reserved word that may be used in a step command. It serves as an abstraction to launch a job with
parallel schedulers like What is Slurm?, What is LSF?, and What is flux?.
How do I use LAUNCHER?
Instead of this:
run:
      cmd: srun -N 1 -n 3 python script.py
Do something like this:
batch:
      type: slurm
run:
      cmd: $(LAUNCHER) python script.py
      nodes: 1
      procs: 3
The arguments the LAUNCHER syntax will use:
procs: The total number of MPI tasks
nodes: The total number of MPI nodes
walltime: The total walltime of the run (hh:mm:ss or mm:ss or ss) (not available in lsf)
cores per task: The number of hardware threads per MPI task
gpus per task: The number of GPUs per MPI task
SLURM specific run flags:
slurm: Verbatim flags only for the srun parallel launch (srun -n <nodes> -n <procs> <slurm>)
FLUX specific run flags:
flux: Verbatim flags for the flux parallel launch (flux mini run <flux>)
LSF specific run flags:
bind: Flag for MPI binding of tasks on a node (default: -b rs)
num resource set: Number of resource sets
launch_distribution: The distribution of resources (default: plane:{procs/nodes})
lsf: Verbatim flags only for the lsf parallel launch (jsrun . . . <lsf>)
68                                                                                 Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
What is level_max_dirs?
level_max_dirs is an optional field that goes under the merlin.samples section of a yaml spec. It caps the number
of sample directories that can be generated at a single level of a study’s sample hierarchy. This is useful for getting
around filesystem constraints when working with massive amounts of data.
Defaults to 25.
What is pgen?
pgen stands for “parameter generator”. It’s a way to override the parameters in the global.parameters spec section,
instead generating them programatically with a python script. Merlin offers the same pgen functionality as Maestro.
See this guide for details on using pgen. It’s a Maestro doc, but the exact same flags can be used in conjunction with
merlin run.
Where can I learn more about merlin?
Check out our paper on arXiv.
1.4 Command line
The merlin executable defines a number of commands to create tasks, launch workers to run the tasks and remove tasks
from the task server. The tasks are communicated to a task server, or broker, that are then requested by workers on an
allocation to run. The celery python module is used to implement the tasks and worker functionality.
1.4.1 Help (merlin --help)
Descriptions of the Merlin commands are outputted when the -h or --help commands are used.
$ merlin [<command name>] --help
1.4.2 Version (merlin --version)
See the version by using the --version or -v flag.
$ merlin --version
1.4.3 Log Level (merlin -lvl debug)
More information, generally pertaining to bugs, can be output by increasing the logging level using the -lvl or
--level argument.
Options for the level argument are: debug, info, warning, error.
$ merlin -lvl debug run <input.yaml>
1.4. Command line                                                                                                    69
Merlin Documentation, Release 1.11.0
1.4.4 Create the Config File (merlin config)
Create a default config file in the ${HOME}/.merlin directory using the config command. This file can then be edited
for your system configuration.
$ merlin config [--task_server]              [--output_dir <dir>] [--broker <rabbitmq|redis>]
The --task_server option will select the appropriate configuration for the given task server. Currently only celery
is implemented.
The --output_dir or -o will output the configuration in the given directory. This file can then be edited and copied
into ${HOME}/.merlin.
The --broker command will write the initial app.yaml config file for a rabbitmq or redis broker. The default is
rabbitmq. The backend will be redis in both cases. The redis backend in the rabbitmq config shows the use on
encryption for the backend.
1.4.5 Generate working examples (merlin example)
If you want to run an example workflow, use Merlin’s merlin example:
$ merlin example list
This will list the available example workflows and a description for each one. To select one:
$ merlin example <example_name>
This will copy the example workflow to the current working directory. It is possible to specify another path to copy to.
$ merlin example <example_name> -p path/to/dir
If the specified directory does not exist Merlin will automatically create it.
This will generate the example workflow at the specified location, ready to be run.
1.4.6 Information (merlin info)
Information about your merlin and python configuration can be printed out by using the info command. This is helpful
for debugging. Included in this command is a server check which will check for server connections. The connection
check will timeout after 60 seconds.
$ merlin info
1.4.7 Monitor (merlin monitor)
Batch submission scripts may not keep the batch allocation alive if there is not a blocking process in the submission
script. The merlin monitor command addresses this by providing a blocking process that checks for tasks in the
queues every (sleep) seconds. When the queues are empty, the blocking process will exit and allow the allocation to
end.
$ merlin monitor <input.yaml> [--steps <steps>] [--vars <VARIABLES=<VARIABLES>>] [--
 ˓→sleep <duration>][--task_server celery]
70                                                                                    Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
Use the --steps option to identify specific steps in the specification that you want to query.
The --vars option will specify desired Merlin variable values to override those found in the specification. The list is
space-delimited and should be given after the input yaml file. Example: --vars LEARN=path/to/new_learn.py
EPOCHS=3
The --sleep argument is the duration in seconds between checks for workers. The default is 60 seconds.
The only currently available option for --task_server is celery, which is the default when this flag is excluded.
The monitor function will check for celery workers for up to 10*(sleep) seconds before monitoring begins. The loop
happens when the queue(s) in the spec contain tasks, but no running workers are detected. This is to protect against a
failed worker launch.
1.4.8 Purging Tasks (merlin purge)
Once the merlin run command succeeds, the tasks are now on the task server waiting to be run by the workers. If you
would like to remove the tasks from the server, then use the purge command.
  Attention: Any tasks reserved by workers will not be purged from the queues. All workers must be first stopped
  so the tasks can be returned to the task server and then they can be purged.
  You probably want to use merlin stop-workers first.
To purge all tasks in all queues defined by the workflow yaml file from the task server, run:
$ merlin purge <input.yaml> [-f] [--steps <steps>] [--vars <VARIABLES=<VARIABLES>>]
This will ask you if you would like to remove the tasks, you can use the -f option if you want to skip this.
If you have different queues in your workflow yaml file, you can choose which queues are purged by using the --steps
argument and giving a space-delimited list of steps.
$ merlin purge <input.yaml> --steps step1 step2
The --vars option will specify desired Merlin variable values to override those found in the specification. The
list is space-delimited and should be given after the input yaml file. Example: --vars QUEUE_NAME=new_queue
EPOCHS=3
1.4.9 Searching for any workers (merlin query-workers)
If you want to see all workers that are currently connected to the task server you can use:
$ merlin query-workers
This will broadcast a command to all connected workers and print the names of any that respond and the queues they’re
attached to. This is useful for interacting with workers, such as via merlin stop-workers --workers.
The --queues option will look for workers associated with the names of the queues you provide here. For example, if
you want to see the names of all workers attached to the queues named demo and merlin you would use:
merlin query-workers --queues demo merlin
The --spec option will query for workers defined in the spec file you provide. For example, if simworker and
nonsimworker are defined in a spec file called example_spec.yaml then to query for these workers you would use:
1.4. Command line                                                                                                  71
Merlin Documentation, Release 1.11.0
merlin query-workers --spec example_spec.yaml
The --workers option will query for workers based on the worker names you provide here. For example, if you wanted
to query a worker named step_1_worker you would use:
merlin query-workers --workers step_1_worker
This flag can also take regular expressions as input. For instance, if you had several workers running but only wanted
to find the workers whose names started with step you would use:
merlin query-workers --workers ^step
1.4.10 Restart the workflow (merlin restart)
To restart a previously started merlin workflow, use the restart command and the path to root of the merlin workspace
that was generated during the previously run workflow. This will define the tasks and queue them on the task server
also called the broker.
$ merlin restart [--local] <path/to/workspace_timestamp>
Merlin currently writes file called MERLIN_FINISHED to the directory of each step that was finished successfully. It
uses this to determine which steps to skip during execution of a workflow.
The --local option will run tasks sequentially in your current shell.
1.4.11 Run the workflow (merlin run)
To run the merlin workflow use the run command and the path to the input yaml file <input.yaml>. This will define
the tasks and queue them on the task server also called the broker.
$ merlin run [--local] <input.yaml> [--vars <VARIABLES=<VARIABLES>>] [--samplesfile
 ˓→<SAMPLES_FILE>] [--dry]
The --local option will run tasks sequentially in your current shell.
The --vars option will specify desired Merlin variable values to override those found in the specification. The list is
space-delimited and should be given after the input yaml file. Example: --vars LEARN=path/to/new_learn.py
EPOCHS=3
The --samplesfile will allow the user to specify a file containing samples. Valid choices: .npy, .csv, .tab. Should
be given after the input yaml file.
The --no-errors option is used for testing, it will silence the errors thrown when flux is not present.
72                                                                                     Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
Dry Run
‘Dry run’ means telling workers to create a study’s workspace and all of its necessary subdirectories and scripts (with
variables expanded) without actually executing the scripts.
To dry-run a workflow, use --dry:
$ merlin run --local --dry <input.yaml>
In a distributed fashion:
$ merlin run --dry <input.yaml> ; merlin run-workers <input.yaml>
You can also specify dry runs from the workflow specification file:
batch:
      dry_run: True
If you wish to execute a workflow after dry-running it, simply use restart.
1.4.12 Run the Workers (merlin run-workers)
The tasks queued on the broker are run by a collection of workers. These workers can be run local in the current
shell or in parallel on a batch allocation. The workers are launched using the run-workers command which reads the
configuration for the worker launch from the <input.yaml> file. The batch and merlin resources section are both used
to configure the worker launch. The top level batch section can be overridden in the merlin workers resource section.
Parallel workers should be scheduled using the system’s batch scheduler. Once the workers are running, tasks from the
broker will be processed.
To launch workers for your workflow:
$ merlin run-workers [--echo] <input.yaml> [--worker-args <worker args>] [--steps
 ˓→<WORKER_STEPS>] [--vars <VARIABLES=<VARIABLES>>]
The --echo option will echo the celery workers run command to stdout and not run any workers.
The --worker-args option will pass the values, in quotes, to the celery workers. Should be given after the input yaml
file.
The --steps option is the specific steps in the input yaml file you want to run the corresponding workers. The default
is ‘all’ steps. Should be given after the input yaml file.
The --vars option will specify desired Merlin variable values to override those found in the specification. The list is
space-delimited and should be given after the input yaml file. Example: --vars LEARN=path/to/new_learn.py
EPOCHS=3
An example of launching a simple celery worker using srun:
$ srun -n 1 celery -A merlin worker -l INFO
A parallel batch allocation launch is configured to run a single worker process per node. This worker process will then
launch a number of worker threads to process the tasks. The number of threads can be configured by the users and will
be the number of parallel jobs that can be run at once on the allocation plus threads for any non-parallel tasks. If there
are 36 cores on a node and all the tasks are single core, the user may want to start 36 threads per node. If the parallel
jobs uses 8 tasks, then the user should run 4 or 5 threads. For the celery workers the number of threads is set using the
--concurrency argument, see the Configuring celery workers section.
A full SLURM batch submission script to run the workflow on 4 nodes is shown below.
1.4. Command line                                                                                                      73
Merlin Documentation, Release 1.11.0
#!/bin/bash
#SBATCH -N 4
#SBATCH -J Merlin
#SBATCH -t 30:00
#SBATCH -p pdebug
#SBATCH --mail-type=ALL
#SBATCH -o merlin_workers_%j.out
# Assumes you are run this in the same dir as the yaml file.
YAML_FILE=input.yaml
# Source the merlin virtualenv
source <path to merlin venv>/bin/activate
# Remove all tasks from the queues for this run.
#merlin purge -f ${YAML_FILE}
# Submit the tasks to the task server
merlin run ${YAML_FILE}
# Print out the workers command
merlin run-workers ${YAML_FILE} --echo
# Run the workers on the allocation
merlin run-workers ${YAML_FILE}
# Delay until the workers cease running
merlin monitor
1.4.13 Status (merlin status)
$ merlin status <input.yaml> [--steps <steps>] [--vars <VARIABLES=<VARIABLES>>] [--csv
 ˓→<csv file>] [--task_server celery]
Use the --steps option to identify specific steps in the specification that you want to query.
The --vars option will specify desired Merlin variable values to override those found in the specification. The list is
space-delimited and should be given after the input yaml file. Example: --vars LEARN=path/to/new_learn.py
EPOCHS=3
The --csv option takes in a filename, to dump status reports to.
The only currently available option for --task_server is celery, which is the default when this flag is excluded.
74                                                                                    Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
1.4.14 Stopping workers (merlin stop-workers)
To send out a stop signal to some or all connected workers, use:
$ merlin stop-workers [--spec <input.yaml>] [--queues <queues>] [--workers <regex>] [--
 ˓→task_server celery]
The default behavior will send a stop to all connected workers across all workflows, having them shutdown softly.
The --spec option targets only workers named in the merlin block of the spec file.
The --queues option allows you to pass in the names of specific queues to stop. For example:
# Stop all workers on these queues, no matter their name
$ merlin stop-workers --queues queue1 queue2
The --workers option allows you to pass in regular expressions of names of workers to stop:
# Stop all workers whose name matches this pattern, no matter the queue
# Note the ".*" convention at the start, per regex
$ merlin stop-workers --workers ".*@my_other_host*"
The only currently available option for --task_server is celery, which is the default when this flag is excluded.
   Attention: If you’ve named workers identically (you shouldn’t) only one might get the signal. In this case, you
   can send it again.
1.4.15 Hosting Local Server (merlin server)
To create a local server for merlin to connect to. Merlin server creates and configures a server on the current directory.
This allows multiple instances of merlin server to exist for different studies or uses.
The init subcommand initalizes a new instance of merlin server.
The status subcommand checks to the status of the merlin server.
The start subcommand starts the merlin server.
The stop subcommand stops the merlin server.
The restart subcommand performs stop command followed by a start command on the merlin server.
The config subcommand edits configurations for the merlin server. There are multiple flags to allow for different
configurations.
     • The -ip IPADDRESS, --ipaddress IPADDRESS option set the binded IP address for merlin server.
     • The -p PORT, --port PORT option set the binded port for merlin server.
     • The -pwd PASSWORD, --password PASSWORD option set the password file for merlin server.
     • The --add-user USER PASSWORD option add a new user for merlin server.
     • The --remove-user REMOVE_USER option remove an exisiting user from merlin server.
     • The -d DIRECTORY, --directory DIRECTORY option set the working directory for merlin server.
     • The -ss SNAPSHOT_SECONDS, --snapshot-seconds SNAPSHOT_SECONDS option set the number of sec-
       onds before each snapshot.
1.4. Command line                                                                                                      75
Merlin Documentation, Release 1.11.0
    • The -sc SNAPSHOT_CHANGES, --snapshot-changes SNAPSHOT_CHANGES option set the number of
       database changes before each snapshot.
    • The -sf SNAPSHOT_FILE, --snapshot-file SNAPSHOT_FILE option set the name of snapshots.
    • The -am APPEND_MODE, --append-mode APPEND_MODE option set the appendonly mode. Options are al-
       ways, everysec, no.
    • The -af APPEND_FILE, --append-file APPEND_FILE option set the filename for server append/change
       file.
More information can be found on Merlin Server
1.5 Workflows
The Merlin package provides a few example workflows. These may be useful in seeing how the software works, and in
designing your own workflow. This section provides documentation on running these Merlin workflow examples.
1.5.1 Overview
List the built-in Merlin workflow examples with merlin example list.
The Merlin team is working on adding a more diverse array of example workflows like these.
In particular, look at the .yaml files within these directories. These are known as Merlin specifications, and are
foundational to determining a workflow.
1.5.2 Get started with the demo ensemble
Merlin provides a demo workflow that highlights some features of the software.
Tip: Have at least two terminals open; one to monitor workers, and the other to provide them tasks.
Create your workflow example:
$ merlin example feature_demo
To run the distributed version of feature_demo, run the following:
$ merlin run feature_demo/feature_demo.yaml
This will queue the tasks to the configured broker. To process the queued tasks, use the run-workers Merlin CLI
command. Adding this command to a parallel batch submission script will launch the workers in parallel.
$ merlin run-workers feature_demo/feature_demo.yaml
76                                                                                 Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
1.6 Workflow Specification
The merlin input file or spec file is separated into several sections. An annotated version is given below.
Note: The Merlin input file is a yaml file and must adhere to yaml syntax. The yaml spec relies on the indentation in
the file.
The input file can take a number of variables, beyond the examples shown here. For a complete list and descriptions of
the variables, see Variables.
####################################
# Description Block (Required)
####################################
# The description block is where the description of the                      study is placed. This
# section is meant primarily for documentation purposes                      so that when a
# specification is passed to other users they can glean                      a general understanding
# of what this study is meant to achieve.
#-------------------------------
# Required keys:
#     name - Name of the study
#     description - Description of what this study does.
#-------------------------------
# NOTE: You can add other keys to this block for custom                      documentation. Merlin
# currently only looks for the required set.
####################################
description:
   description: Run a scan through Merlin
   name: MERLIN
####################################
# Batch Block (Required)
####################################
# The batch system to use for each allocation
#-------------------------------
# Required keys:
#     type - The scheduler type to use (local|slurm|flux|lsf)
#     bank - The allocation bank
#     queue - The batch queue
####################################
batch:
    type: flux
    bank: testbank
    queue: pbatch
    flux_path: <optional path to flux bin>
    flux_start_opts: <optional flux start options>
    flux_exec: <optional, flux exec command to launch workers on
                                all nodes if using flux and flux_exec_workers is True
                                (flux exec) >
    flux_exec_workers: <optional, flux argument to launch workers on
                                all nodes. (True)>
    launch_pre: <Any configuration needed before the srun or jsrun launch>
    launch_args: <Optional extra arguments for the parallel launch command>
                                                                                                      (continues on next page)
1.6. Workflow Specification                                                                                                77
Merlin Documentation, Release 1.11.0
                                                                        (continued from previous page)
   worker_launch: <Override the parallel launch defined in merlin>
   shell: <the interpreter to use for the script after the shebang>
           # e.g. /bin/bash, /bin/tcsh, python, /usr/bin/env perl, etc.
   nodes: <num nodes> # The number of nodes to use for all workers
                         This can be overridden in the workers config.
                         If this is unset the number of nodes will be
                         queried from the environment, failing that, the
                         number of nodes will be set to 1.
   walltime: The total walltime of the batch allocation (hh:mm:ss or mm:ss or ss)
#####################################
# Environment Block
####################################
# The environment block is where items describing the study's environment are
# defined. This includes static information that the study needs to know about
# and dependencies that the workflow requires for execution.
#-------------------------------
# NOTE: This block isn't strictly required as a study may not depend on anything.
########################################################################
env:
  #-------------------------------
  # Variables
  #-------------------------------
  # Values that the workflow substitutes into steps and are similar in
  # concept to Unix environment variables. These variables are not dependent
  # on values in the environment and so are more portable.
  #
  # Note that variables defined here can alter the runtime shell
  # variable definitions.
  # Do not define a variable named "shell" here.
  #-------------------------------
  variables:
    # Set a custom output path for the study workspace. This path is where
    # Merlin will place all temporary files, state files, and any output.
    # The resulting path is usually a timestamped folder within OUTPUT_PATH
    # and in this case would be
    # './sample_output/merlin/merlin_sample1_<timestamp>'.
    # NOTE: If not specified,
    # OUTPUT_PATH is assumed to be the path where Merlin was launched from.
    OUTPUT_PATH: ./sample_output/merlin # OUTPUT_PATH is a keyword
                                         # variable that Merlin looks for
                                         # to replace with the study
                                         # directory created for the
                                         # ensemble
####################################
# Study Block (Required)
####################################
# The study block is where the steps in the workflow are defined. This section
# of the specification represents the unexpanded set of tasks that the study
# is composed of.
                                                                               (continues on next page)
78                                                               Chapter 1. Merlin Overview
                                                        Merlin Documentation, Release 1.11.0
                                                                       (continued from previous page)
#
#
#  A description of what gets turned into tasks and what type of task
#  would be a good addition
#
#  study lists the various steps, each of which has these fields
#  name: step name
#  description: what the step does
#  run:
#    cmd: the command to run for multilines use cmd: |
#         The $(LAUNCHER) macro can be used to substitute a parallel launcher
#         based on the batch:type:.
#         It will use the nodes and procs values for the task.
#    task_queue: the queue to assign the step to (optional. default: merlin)
#    shell: the shell to use for the command (eg /bin/bash /usr/bin/env python)
#           (optional. default: /bin/bash)
#    depends: a list of steps this step depends upon (ie parents)
#    procs: The total number of MPI tasks
#    nodes: The total number of MPI nodes
#    walltime: The total walltime of the run (hh:mm:ss, mm:ss or ss) (not available in␣
 ˓→lsf)
#    cores per task: The number of hardware threads per MPI task
#    gpus per task: The number of GPUs per MPI task
#    SLURM specific run flags:
#    slurm: Verbatim flags only for the srun parallel launch (srun -n <nodes> -n <procs>
 ˓→<slurm>)
#    FLUX specific run flags:
#    flux: Verbatim flags for the flux parallel launch (flux mini run <flux>)
#    LSF specific run flags:
#    bind: Flag for MPI binding of tasks on a node
#    num resource set: Number of resource sets
#    launch_distribution : The distribution of resources (default: plane:{procs/nodes})
#    exit_on_error: Flag to exit on error (default: 1)
#    lsf: Verbatim flags only for the lsf parallel launch (jsrun ... <lsf>
#######################################################################
  study:
   - name: runs1
     description: Run on alloc1
     run:
      cmd: $(LAUNCHER) echo "$(VAR1) $(VAR2)" > simrun.out
      nodes: 1
      procs: 1
      task_queue: queue1
      shell: /bin/bash
   - name: post-process
     description: Post-Process runs on alloc1
     run:
       cmd: |
         cd $(runs1.workspace)/$(MERLIN_SAMPLE_PATH)
         <post-process>
       nodes: 1
                                                                              (continues on next page)
1.6. Workflow Specification                                                                        79
Merlin Documentation, Release 1.11.0
                                                                        (continued from previous page)
       procs: 1
       depends: [runs1]
       task_queue: queue1
  - name: runs2
    description: Run on alloc2
    run:
       cmd: |
         touch learnrun.out
         $(LAUNCHER) echo "$(VAR1) $(VAR2)" >> learnrun.out
         exit $(MERLIN_RETRY) # some syntax to send a retry error code
       nodes: 1
       procs: 1
       task_queue: lqueue
       max_retries: 3     # maximum number of retries
       retry_delay: 10    # delay retry for N seconds (default 1)
       batch:
         type: <override the default batch type>
  - name: monitor
    description: Monitor on alloc1
    run:
       cmd: date > monitor.out
       nodes: 1
       procs: 1
       task_queue: mqueue
####################################
# Parameter Block (Required)
####################################
# The parameter block contains all the things we'd like to vary in the study.
# Currently, there are two modes of operating in the specification:
# 1. If a parameter block is specified, the study is expanded and considered a
#   parameterized study.
# 2. If a parameter block is not specified, the study is treated as linear and
#    the resulting study is not expanded.
#
# There are three keys per parameter:
# 1. A list of values that the parameter takes.
# 2. A label that represents a "pretty printed" version of the parameter. The
#    parameter values is specified by the '%%' moniker (for example, for SIZE --
#    when SIZE is equal to 10, the label will be 'SIZE.10'). To access the label
#    for SIZE, for example, the token '$(SIZE.label)' is used.
#    Labels can take one of two forms: A single string with the '%%' marker or
#    a list of per value labels (must be the same length as the list of values).
#
# NOTE: A specified parameter does not necessarily have to be used in every step
# or at all. If a parameter is specified and not used, it simply will not be
# factored into expansion or the naming of expanded steps or their workspaces.
# NOTE: You can also specify custom generation of parameters using a Python
# file containing the definition of a function as follows:
#
                                                                               (continues on next page)
80                                                                Chapter 1. Merlin Overview
                                                         Merlin Documentation, Release 1.11.0
                                                                        (continued from previous page)
# 'def get_custom_generator():'
#
# The 'get_custom_generator' function is required to return a ParameterGenerator
# instance populated with custom filled values. In order to use the file, simply
# call Merlin using 'merlin run <specification path>'.
########################################################################
global.parameters:
   STUDY:
     label: STUDY.%%
     values: [MERLIN1, MERLIN2]
   SIZE:
      values : [10, 20]
      label    : SIZE.%%
   ITERATIONS:
      values : [10, 20]
      label    : ITER.%%
####################################
# Merlin Block (Required)
####################################
# The merlin specific block will add any configuration to
# the DAG created by the study description.
# including task server config, data management and sample definitions.
#
# merlin will replace all SPECROOT instances with the directory where
# the input yaml was run.
#######################################################################
merlin:
   ####################################
   # Resource definitions
   #
   # Define the task server configuration and workers to run the tasks.
   #
   ####################################
   resources:
     task_server: celery
     # Flag to determine if multiple workers can pull tasks
     # from overlapping queues. (default = False)
     overlap: False
     # Customize workers. Workers can have any user-defined name (e.g., simworkers,␣
 ˓→learnworkers).
     workers:
         simworkers:
              args: <celery worker args> <optional>
              steps: [runs1, post-process, monitor] # [all] when steps is omitted
              nodes: <Number of nodes for this worker or batch num nodes>
              # A list of machines to run the given steps can be specified
              # in the machines keyword. <optional>
              # A full OUTPUT_PATH and the steps argument are required
                                                                               (continues on next page)
1.6. Workflow Specification                                                                         81
Merlin Documentation, Release 1.11.0
                                                                         (continued from previous page)
              # when using this option. Currently all machines in the
              # list must have access to the OUTPUT_PATH.
              machines: [host1, host2]
         learnworkers:
              args: <celery worker args> <optional>
              steps: [runs2]
              nodes: <Number of nodes for this worker or batch num nodes>
              # An optional batch section in the worker can override the
              # main batch config. This is useful if other workers are running
              # flux, but some component of the workflow requires the native
              # scheduler or cannot run under flux. Another possibility is to
              # have the default type as local and workers needed for flux or
              # slurm steps.
              batch:
                 type: local
              machines: [host3]
   ###################################################
   # Sample definitions
   #
   # samples file can be one of
   #    .npy (numpy binary)
   #    .csv (comma delimited: '#' = comment line)
   #    .tab (tab/space delimited: '#' = comment line)
   ###################################################
   samples:
     column_labels: [VAR1, VAR2]
     file: $(SPECROOT)/samples.npy
     generate:
       cmd: |
       python $(SPECROOT)/make_samples.py -dims 2 -n 10 -outfile=$(INPUT_PATH)/samples.
 ˓→npy "[(1.3, 1.3, 'linear'), (3.3, 3.3, 'linear')]"
     level_max_dirs: 25
####################################
# User Block (Optional)
####################################
# The user block allows other variables in the workflow file to be propagated
# through to the workflow (including in variables .partial.yaml and .expanded.yaml).
# User block uses yaml anchors, which defines a chunk of configuration and use
# their alias to refer to that specific chunk of configuration elsewhere.
#######################################################################
user:
   study:
       run:
            hello: &hello_run
                cmd: |
                  python3 $(HELLO) -outfile hello_world_output_$(MERLIN_SAMPLE_ID).json
 ˓→$(X0) $(X1) $(X2)
                max_retries: 1
            collect: &collect_run
                                                                                (continues on next page)
82                                                                Chapter 1. Merlin Overview
                                                        Merlin Documentation, Release 1.11.0
                                                                       (continued from previous page)
               cmd: |
                 echo $(MERLIN_GLOB_PATH)
                 echo $(hello.workspace)
                 ls $(hello.workspace)/X2.$(X2)/$(MERLIN_GLOB_PATH)/hello_world_output_*.
˓→json > files_to_collect.txt
                 spellbook collect -outfile results.json -instring "$(cat files_to_
˓→collect.txt)"
           translate: &translate_run
               cmd: spellbook translate -input $(collect.workspace)/results.json -output␣
˓→results.npz -schema $(FEATURES)
           learn: &learn_run
               cmd: spellbook learn -infile $(translate.workspace)/results.npz
           make_samples: &make_samples_run
               cmd: spellbook make-samples -n $(N_NEW) -sample_type grid -outfile grid_
˓→$(N_NEW).npy
           predict: &predict_run
               cmd: spellbook predict -infile $(make_new_samples.workspace)/grid_$(N_NEW).
˓→npy -outfile prediction_$(N_NEW).npy -reg $(learn.workspace)/random_forest_reg.pkl
           verify: &verify_run
               cmd: |
                 if [[ -f $(learn.workspace)/random_forest_reg.pkl && -f $(predict.
˓→workspace)/prediction_$(N_NEW).npy ]]
                 then
                     touch FINISHED
                     exit $(MERLIN_SUCCESS)
                 else
                     exit $(MERLIN_SOFT_FAIL)
                 fi
  python3:
      run: &python3_run
           cmd: |
             print("OMG is this in python?")
             print("Variable X2 is $(X2)")
           shell: /usr/bin/env python3
  python2:
      run: &python2_run
           cmd: |
             print "OMG is this in python2? Change is bad."
             print "Variable X2 is $(X2)"
           shell: /usr/bin/env python2
1.6. Workflow Specification                                                                       83
Merlin Documentation, Release 1.11.0
1.7 Configuration
This section provides documentation for configuring Merlin’s connections with task servers and results backends.
1.7.1 Merlin server configuration
Merlin works best configuring celery to run with a RabbitMQ broker and a redis backend. Merlin uses celery chords
which require a results backend be configured. The Amqp (rpc Rabbitmq) server does not support chords but the Redis,
Database, Memcached and more, support chords.
Merlin’s configuration is controlled by an app.yaml file, such as the one below:
celery:
     # directory where Merlin looks for the following:
     # mysql-ca-cert.pem rabbit-client-cert.pem rabbit-client-key.pem redis.pass
     certs: /path/to/celery/config
broker:
     # can be rabbitmq, redis, rediss, or redis+sock
     name: rabbitmq
     #username: # defaults to your username unless changed here
     password: ~/.merlin/rabbit-password
     # server URL
     server: server.domain.com
     ### for rabbitmq connections ###
     #vhost: # defaults to your username unless changed here
     ### for redis+sock connections ###
     #socketname: the socket name your redis connection can be found on.
     #path: The path to the socket.
     ### for redis/rediss connections ###
     #port: The port number redis is listening on (default 6379)
     #db_num: The data base number to connect to.
     # ssl security
     #keyfile: /var/ssl/private/client-key.pem
     #certfile: /var/ssl/amqp-server-cert.pem
     #ca_certs: /var/ssl/myca.pem
     # This is optional and can be required, optional or none
     # (required is the default)
     #cert_reqs: required
results_backend:
     # Can be redis,rediss, mysql, db+ or memcached server
     # Only a few of these are directly configured by merlin
     name: redis
     dbname: dbname
     username: username
                                                                                                 (continues on next page)
84                                                                                 Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
                                                                                              (continued from previous page)
      # name of file where redis password is stored.
      password: redis.pass
      server: server.domain.com
      # merlin will generate this key if it does not exist yet,
      # and will use it to encrypt all data over the wire to
      # your redis server.
      encryption_key: ~/.merlin/encrypt_data_key
      port: 6379
      db_num: 0
      # ssl security
      #keyfile: /var/ssl/private/client-key.pem
      #certfile: /var/ssl/amqp-server-cert.pem
      #ca_certs: /var/ssl/myca.pem
      # This is optional and can be required, optional or none
      # (required is the default)
      #cert_reqs: required
The default location for the app.yaml is in the merlin repo under the merlin/config directory. This default can be
overridden by files in one of two other locations. The current working directory is first checked for the app.yaml file,
then the user’s ~/.merlin directory is checked.
broker/name: can be rabbitmq, redis, or redis+sock. As their names imply, rabbitmq will use RabbitMQ as
a task broker (preferred for multi user configurations), redis will use redis as a task broker, and redis+sock will
connect to a redis task broker using a socket.
1.7.2 Broker: rabbitmq, amqps, amqp
Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file.
If the port argument is not defined, the default rabbitmq TLS port, 5671, will be used. See the Security with RabbitMQ
section for more info about security with this broker. When the broker is amqp, the default port will be 5672.
The prototype url for this configuration is:
{conn}://{username}:{password}@{server}:{port}/{vhost}
Here conn is amqps (with ssl) when name is rabbitmq or amqps and amqp (without ssl) when name is amqp.
broker:
    name: rabbitmq
    #username: # defaults to your username unless changed here
    password: ~/.merlin/rabbit-password
    # server URL
    server: server.domain.com
    #vhost: # defaults to your username unless changed here
1.7. Configuration                                                                                                       85
Merlin Documentation, Release 1.11.0
1.7.3 Broker: redis
Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file.
The prototype url for this configuration is:
redis://:{password}@{server}:{port}/{db_num}
broker:
  name: redis
  server: localhost
  port: 6379
1.7.4 Broker: rediss
Newer versions of Redis (version 6 or greater) can be configured with ssl. The rediss name is used to enable this
support. See the Security with redis section for more info.
The prototype url for this configuration is:
rediss://:{password}@{server}:{port}/{db_num}
broker:
  name: rediss
  server: localhost
  port: 6379
1.7.5 Broker: redis+socket
Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file.
The prototype url for this configuration is:
redis+socket://{path}?virtual_host={db_num}
broker:
  name: redis+socket
  path: /tmp/username/redis.sock
  db_num: 0
86                                                                                   Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
1.7.6 Broker: url
A url option is available to specify the broker connection url, in this case the server name is ignored. The url must
include all the entire connection url except the ssl if the broker name is recognized by the ssl processing system.
Currently the ssl system will only configure the Rabbitmq and Redis servers.
The prototype url for this configuration is:
{url}
broker:
   url: redis://localhost:6379/0
1.7.7 Broker: Security
Security with RabbitMQ
Merlin can only be configured to communicate with RabbitMQ over an SSL connection and does not permit use of a
RabbitMQ server configured without SSL. The default value of the broker_use_ssl keyword is True. The keys can be
given in the broker config as show below.
broker:
   # can be redis, redis+sock, or rabbitmq
   name: rabbitmq
   #username: # defaults to your username unless changed here
   password: ~/.merlin/rabbit-password
   # server URL
   server: server.domain.com
   ### for rabbitmq, redis+sock connections ###
   #vhost: # defaults to your username unless changed here
   # ssl security
   keyfile: /var/ssl/private/client-key.pem
   certfile: /var/ssl/amqp-server-cert.pem
   ca_certs: /var/ssl/myca.pem
   # This is optional and can be required, optional or none
   # (required is the default)
   cert_reqs: required
This results in a value for broker_use_ssl given below:
broker_use_ssl = {
'keyfile': '/var/ssl/private/client-key.pem',
'certfile': '/var/ssl/amqp-server-cert.pem',
'ca_certs': '/var/ssl/myca.pem',
'cert_reqs': ssl.CERT_REQUIRED
}
1.7. Configuration                                                                                                87
Merlin Documentation, Release 1.11.0
Security with redis
The same ssl config and resulting ssl_use_broker can be used with the rediss:// url when using a redis server version
6 or greater with ssl.
broker:
   name: rediss
   #username: # defaults to your username unless changed here
   # server URL
   server: server.domain.com
   port: 6379
   db_num: 0
   # ssl security
   keyfile: /var/ssl/private/client-key.pem
   certfile: /var/ssl/amqp-server-cert.pem
   ca_certs: /var/ssl/myca.pem
   # This is optional and can be required, optional or none
   # (required is the default)
   cert_reqs: required
The resulting broker_use_ssl configuration for a rediss server is given below.
broker_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}
1.7.8 Results backend: redis
Merlin constructs the following connection string from relevant options in the results_backend section of the app.yaml
file.
The prototype url for this configuration is:
redis://:{password}{server}:{port}/{db_num}
results_backend:
   name: redis
   server: localhost
   port: 6379
88                                                                                    Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
1.7.9 Results backend: rediss
Newer versions of Redis (version 6 or greater) can be configured with ssl. The rediss name is used to enable this
support. See the Security with redis section for more info.
The prototype url for this configuration is:
rediss://:{password}{server}:{port}/{db_num}
results_backend:
   name: rediss
   server: localhost
   port: 6379
1.7.10 Results backend: url
A url option is available to specify the results connection url, in this case the server name is ignored. The url must
include the entire connection url including the ssl configuration.
The prototype url for this configuration is:
{url}
results_backend:
   url: redis://localhost:6379/0
The url option can also be used to define a server that is not explicitly handled by the merlin configuration system.
results_backend:
   url: db+postgresql://scott:tiger@localhost/mydatabase
Resolving password fields
The results_backend/password is interpreted in the following way. First, it is treated as an absolute path to a file
containing your backend password. If that path doesn’t exist, it then looks for a file of that name under the directory
defined under celery/certs. If that file doesn’t exist, it then looks treats results_backend/password as the
password itself.
The broker/password is simply the full path to a file containing your password for the user defined by broker/
username.
1.7. Configuration                                                                                                  89
Merlin Documentation, Release 1.11.0
1.7.11 Results backend: Security
Security with redis
Redis versions less than 6 do not natively support multiple users or SSL. We address security concerns here by redefining
the core celery routine that communicates with redis to encrypt all data it sends to redis and then decrypt anything it
receives. Each user should have their own encryption key as defined by results_backend/encryption_key in the
app.yaml file. Merlin will generate a key if that key does not yet exist.
Redis versions 6 or greater can use the ssl keys as in the broker section. The ssl config with redis (rediss) in the results
backend is the placed in the redis_backend_use_ssl celery argument. The values in this argument are the same as
the broker.
redis_backend_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}
1.8 Variables
There are a number of variables which can be placed in a merlin input .yaml file that can control workflow execution,
such as via string expansion and control flow.
Note: Only user variables and OUTPUT_PATH may be reassigned or overridden from the command line.
1.8.1 Directory structure context
The directory structure of merlin output looks like this:
SPECROOT
     <spec.yaml>
...
OUTPUT_PATH
     MERLIN_WORKSPACE
           MERLIN_INFO
                 <name>.orig.yaml
                 <name>.partial.yaml
                 <name>.expanded.yaml
           <step_name>.workspace
           WORKSPACE
90                                                                                        Chapter 1. Merlin Overview
               Merlin Documentation, Release 1.11.0
1.8. Variables                                   91
Merlin Documentation, Release 1.11.0
1.8.2 Reserved variables
                 Table 6: Study variables that Merlin uses. May be referenced within a
                 specification file, but not reassigned or overridden.
 Variable                 Description                                                Example Expansion
 $(SPECROOT)              Directory path of the specification file.
                                                                                     /globalfs/user/
                                                                                      ˓→merlin_workflows
 $(OUTPUT_PATH)           Directory path the study output will be written to. If
                          not defined will default to the current working directory. ./studies
                          May be reassigned or overridden.
 $(MERLIN_TIMESTAMP)      The time a study began. May be used as a unique iden-
                          tifier.                                                    "YYYYMMDD-HHMMSS"
 $(MERLIN_WORKSPACE)      Output directory generated by a study at OUTPUT_PATH.
                          Ends with MERLIN_TIMESTAMP.                                $(OUTPUT_PATH)/
                                                                                      ˓→ensemble_name_
                                                                                      ˓→$(MERLIN_
                                                                                      ˓→TIMESTAMP)
 $(WORKSPACE)             The workspace directory for a single step.
                                                                                     $(OUTPUT_PATH)/
                                                                                      ˓→ensemble_name_
                                                                                      ˓→$(MERLIN_
                                                                                      ˓→TIMESTAMP)/step_
                                                                                      ˓→name/``
 $(MERLIN_INFO)           Directory within MERLIN_WORKSPACE that holds the
                          provenance specs and sample generation results. Com-       $(MERLIN_WORKSPACE)/
                          monly used to hold samples.npy.                             ˓→merlin_info/
 $(MERLIN_SAMPLE_ID)      Sample index in an ensemble
                                                                                     0 1 2 3
 $(MERLIN_SAMPLE_PATH) Path in the sample directory tree to a sample’s directory,
                          i.e. where the task is actually run.                       /0/0/0/ /0/0/1/ /0/
                                                                                      ˓→0/2/ /0/0/3/
 $(MERLIN_GLOB_PATH)      All of the directories in a simulation tree as a glob (*)
                          string                                                     /*/*/*/*
 $(MERLIN_PATHS_ALL)      A space delimited string of all of the paths; can be used
                          as is in bash for loop for instance with:                  0/0/0
                          for path in $(MERLIN_PATHS_ALL)                            0/0/1
                              do                                                     0/0/2
                                  ls $path                                           0/0/3
                              done
 $(MERLIN_SAMPLE_VECTOR)Vector of merlin sample values
                                                                                     $(SAMPLE_COLUMN_1)
                                                                                      ˓→$(SAMPLE_COLUMN_
                                                                                      ˓→2) ...
 $(MERLIN_SAMPLE_NAMES) Names of merlin sample values
92                                                                                Chapter 1. Merlin Overview
                                                                                     SAMPLE_COLUMN_1␣
                                                                                      ˓→SAMPLE_COLUMN_2 ..
                                                                                      ˓→.
                                                                             Merlin Documentation, Release 1.11.0
The LAUNCHER and VLAUNCHER Variables
$(LAUNCHER) is a special case of a reserved variable since it’s value can be changed. It serves as an abstraction to
launch a job with parallel schedulers like slurm, lsf , and flux and it can be used within a step command. For example,
say we start with this run cmd inside our step:
run:
      cmd: srun -N 1 -n 3 python script.py
We can modify this to use the $(LAUNCHER) variable like so:
batch:
      type: slurm
run:
      cmd: $(LAUNCHER) python script.py
      nodes: 1
      procs: 3
In other words, the $(LAUNCHER) variable would become srun -N 1 -n 3.
Similarly, the $(VLAUNCHER) variable behaves similarly to the $(LAUNCHER) variable. The key distinction lies in its
source of information. Instead of drawing certain configuration options from the run section of a step, it retrieves spe-
cific shell variables. These shell variables are automatically generated by Merlin when you include the $(VLAUNCHER)
variable in a step command, but they can also be customized by the user. Currently, the following shell variables are:
                                             Table 7: VLAUNCHER Variables
  Variable                       Description                                                  Default
  ${MERLIN_NODES}                The number of nodes                                          1
  ${MERLIN_PROCS}                The number of tasks/procs                                    1
  ${MERLIN_CORES}                The number of cores per task/proc                            1
  ${MERLIN_GPUS}                 The number of gpus per task/proc                             0
Let’s say we have the following defined in our yaml file:
batch:
      type: flux
run:
      cmd: |
         MERLIN_NODES=4
         MERLIN_PROCS=2
         MERLIN_CORES=8
         MERLIN_GPUS=2
         $(VLAUNCHER) python script.py
The $(VLAUNCHER) variable would be substituted to flux run -N 4 -n 2 -c 8 -g 2.
1.8. Variables                                                                                                        93
Merlin Documentation, Release 1.11.0
1.8.3 User variables
Variables defined by a specification file in the env section, as in this example:
env:
        variables:
             ID: 42
             EXAMPLE_VAR:        hello
As long as they’re defined in order, you can nest user variables like this:
env:
        variables:
             EXAMPLE_VAR:        hello
             WORKER_NAME: $(EXAMPLE_VAR)_worker
Like all other Merlin variables, user variables may be used anywhere (as a yaml key or value) within a specification as
below:
cmd: echo "$(EXAMPLE_VAR), world!"
...
$(WORKER_NAME):
        args: ...
If you want to programmatically define the study name, you can include variables in the description.name field as
long as it makes a valid filename:
description:
        name: my_$(EXAMPLE_VAR)_study_$(ID)
        description: example of programmatic study name
The above would produce a study called my_hello_study_42.
1.8.4 Environment variables
Merlin expands Unix environment variables for you. The values of the user variables below would be expanded:
env:
        variables:
             MY_HOME: ~/
             MY_PATH: $PATH
             USERNAME: ${USER}
However, Merlin leaves environment variables found in shell scripts (think cmd and restart) alone. So this step:
- name: step1
     description: an example
     run:
        cmd: echo $PATH ; echo $(MY_PATH)
. . . would be expanded as:
94                                                                                  Chapter 1. Merlin Overview
                                                       Merlin Documentation, Release 1.11.0
- name: step1
  description: an example
  run:
     cmd: echo $PATH ; echo /an/example/:/path/string/
1.8. Variables                                                                           95
Merlin Documentation, Release 1.11.0
96                                   Chapter 1. Merlin Overview
                                                                    Merlin Documentation, Release 1.11.0
1.8.5 Step return variables
                          Table 8: Special return code variables for task steps.
 Variable               Description                                                Example Usage
 $(MERLIN_SUCCESS)      This step was successful. Keep going to the next task.
                        Default step behavior if no exit code given.               echo "hello, world!"
                                                                                   exit $(MERLIN_
                                                                                    ˓→SUCCESS)
 $(MERLIN_RESTART)      Run this step’s restart command, or re-run cmd if
                        restart is absent. The default maximum number of           run:
                        retries+restarts for any given step is 30. You can over-      cmd: |
                        ride this by adding a max_retries field under the run            touch my_file.
                        field in the specification. Issues a warning. Default will  ˓→txt
                        retry in 1 second. To override the delay time, specify           echo "hi mom!"␣
                        retry_delay.                                                ˓→>> my_file.txt
                                                                                         exit $(MERLIN_
                                                                                    ˓→RESTART)
                                                                                      restart: |
                                                                                         echo "bye, mom!
                                                                                    ˓→" >> my_file.txt
                                                                                      max_retries: 23
                                                                                      retry_delay: 10
 $(MERLIN_RETRY)        Retry this step’s cmd command. The default maximum
                        number of retries for any given step is 30. You can over-  run:
                        ride this by adding a max_retries field under the run         cmd: |
                        field in the specification. Issues a warning. Default will       touch my_file.
                        retry in 1 second. To override the delay time, specify      ˓→txt
                        retry_delay.                                                     echo "hi mom!"␣
                                                                                    ˓→>> my_file.txt
                                                                                         exit $(MERLIN_
                                                                                    ˓→RETRY)
                                                                                      max_retries: 23
                                                                                      retry_delay: 10
 $(MERLIN_SOFT_FAIL)    Mark this step as a failure, note in the warning log but
                        keep going. Unknown return codes get translated to soft    echo "Uh-oh, this␣
                        fails, so that they can be logged.                          ˓→sample didn't work
                                                                                    ˓→"
                                                                                   exit $(MERLIN_SOFT_
                                                                                    ˓→FAIL)
 $(MERLIN_HARD_FAIL)    Something went terribly wrong and I need to stop the
                        whole workflow. Raises a HardFailException and             echo "Oh no, we've␣
                        stops all workers connected to that step. Workers will      ˓→created skynet!␣
                        stop after a 60 second delay to allow the step to be ac-    ˓→Abort!"
                        knowledged by the server.                                  exit $(MERLIN_HARD_
                                                                                    ˓→FAIL)
                        Note: Workers in isolated parts of the workflow not
                        consuming from the bad step will continue. You can stop
                        all workers with $(MERLIN_STOP_WORKERS).
 $(MERLIN_STOP_WORKERS) Launch a task to stop all active workers. To allow the
1.8. Variables          current task to finish and acknowledge the results to the  # send a signal to␣ 97
                        server, will happen in 60 seconds.                          ˓→all workers to␣
                                                                                    ˓→stop
                                                                                   exit $(MERLIN_STOP_
Merlin Documentation, Release 1.11.0
1.9 Merlin Server
The merlin server command allows users easy access to containerized broker and results servers for merlin workflows.
This allows users to run merlin without a dedicated external server.
The main configuration will be stored in the subdirectory called “server/” by default in the main merlin configuration
“~/.merlin”. However different server images can be created for different use cases or studies just by simplying creating
a new directory to store local configuration files for merlin server instances.
Below is an example of how merlin server can be utilized.
First create and navigate into a directory to store your local merlin configuration for a specific use case or study.
mkdir study1/
cd study1/
Afterwards you can instantiate merlin server in this directory by running
merlin server init
A main server configuration will be created in the ~/.merlin/server and a local configuration will be created in a subdi-
rectory called “merlin_server/”
We should expect the following files in each directory
~/study1$ ls ~/.merlin/server/
docker.yaml merlin_server.yaml                podman.yaml       singularity.yaml
~/study1$ ls
merlin_server
~/study1$ ls merlin_server/
redis.conf redis_latest.sif
The main configuration in “~/.merlin/server” deals with defaults and technical commands that might be used for setting
up the merlin server local configuration and its containers. Each container has their own configuration file to allow users
to be able to switch between different containerized services freely.
The local configuration “merlin_server” folder contains configuration files specific to a certain use case or run. In the
case above you can see that we have a redis singularity container called “redis_latest.sif” with the redis configuration
file called “redis.conf”. This redis configuration will allow the user to configurate redis to their specified needs without
have to manage or edit the redis container. When the server is run this configuration will be dynamically read, so
settings can be changed between runs if needed.
Once the merlin server has been initialized in the local directory the user will be allowed to run other merlin server
commands such as “run, status, stop” to interact with the merlin server. A detailed list of commands can be found in
the Merlin Server Commands page.
Note: Running “merlin server init” again will NOT override any exisiting configuration that the users might have
set or edited. By running this command again any missing files will be created for the users with exisiting defaults.
HOWEVER it is highly advised that users back up their configuration in case an error occurs where configuration files
are overriden.
98                                                                                       Chapter 1. Merlin Overview
                                                                        Merlin Documentation, Release 1.11.0
1.9.1 Merlin Server Configuration
Below are a sample list of configurations for the merlin server command
Main Configuration ~/.merlin/server/
merlin_server.yaml
container:
      # Select the format for the recipe e.g. singularity, docker, podman (currently␣
 ˓→singularity is the only working option.)
      format: singularity
      # The image name
      image: redis_latest.sif
      # The url to pull the image from
      url: docker://redis
      # The config file
      config: redis.conf
      # Subdirectory name to store configurations Default: merlin_server/
      config_dir: merlin_server/
      # Process file containing information regarding the redis process
      pfile: merlin_server.pf
process:
      # Command for determining the process of the command
      status: pgrep -P {pid} #ps -e | grep {pid}
      # Command for killing process
      kill: kill {pid}
singularity.yaml
singularity:
      command: singularity
      # init_command: \{command} .. (optional or default)
      run_command: \{command} run {image} {config}
      stop_command: kill # \{command} (optional or kill default)
      pull_command: \{command} pull {image} {url}
Local Configuration merlin_server/
redis.conf
bind 127.0.0.1 -::1
protected-mode yes
port 6379
logfile ""
dir ./
...
see documentation on redis configuration here for more detail
merlin_server.pf
1.9. Merlin Server                                                                                        99
Merlin Documentation, Release 1.11.0
bits: '64'
commit: '00000000'
hostname: ubuntu
image_pid: '1111'
mode: standalone
modified: '0'
parent_pid: 1112
port: '6379'
version: 6.2.6
1.9.2 Merlin Server Commands
Merlin server has a list of commands for interacting with the broker and results server. These commands allow the user
to manage and monitor the exisiting server and create instances of servers if needed.
Initializing Merlin Server (merlin server init)
The merlin server init command creates configurations for merlin server commands.
A main merlin sever configuration subdirectory is created in “~/.merlin/server” which contains configuration for local
merlin configuration, and configurations for different containerized services that merlin server supports, which includes
singularity (docker and podman implemented in the future).
A local merlin server configuration subdirectory called “merlin_server/” will also be created when this command is
run. This will contain a container for merlin server and associated configuration files that might be used to start the
server. For example, for a redis server a “redis.conf” will contain settings which will be dynamically loaded when the
redis server is run. This local configuration will also contain information about currently running containers as well.
Note: If there is an exisiting subdirectory containing a merlin server configuration then only missing files will be
replaced. However it is recommended that users backup their local configurations.
Checking Merlin Server Status (merlin server status)
Displays the current status of the merlin server.
Starting up a Merlin Server (merlin server start)
Starts the container located in the local merlin server configuration.
Note: If this command seems to hang and never release control back to you, follow these steps:
   1. Kill the command with Ctrl+C
   2. Run either export LC_ALL="C.UTF-8" or export LC_ALL="C"
   3. Re-run the merlin server start command
100                                                                                     Chapter 1. Merlin Overview
                                                                        Merlin Documentation, Release 1.11.0
Stopping an exisiting Merlin Server (merlin server stop)
Stop any exisiting container being managed and monitored by merlin server.
Restarting a Merlin Server instance (merlin server restart)
Restarting an existing container that is being managed and monitored by merlin server.
Configurating Merlin Server instance (merlin server config)
Place holder for information regarding merlin server config command
Possible Flags
-ip IPADDRESS, --ipaddress IPADDRESS
                                Set the binded IP address for the merlin server
                                container. (default: None)
-p PORT, --port PORT Set the binded port for the merlin server container.
                                (default: None)
-pwd PASSWORD, --password PASSWORD
                                Set the password file to be used for merlin server
                                container. (default: None)
--add-user ADD_USER ADD_USER
                                Create a new user for merlin server instance. (Provide
                                both username and password) (default: None)
--remove-user REMOVE_USER
                                Remove an exisiting user. (default: None)
-d DIRECTORY, --directory DIRECTORY
                                Set the working directory of the merlin server
                                container. (default: None)
-ss SNAPSHOT_SECONDS, --snapshot-seconds SNAPSHOT_SECONDS
                                Set the number of seconds merlin server waits before
                                checking if a snapshot is needed. (default: None)
-sc SNAPSHOT_CHANGES, --snapshot-changes SNAPSHOT_CHANGES
                                Set the number of changes that are required to be made
                                to the merlin server before a snapshot is made.
                                (default: None)
-sf SNAPSHOT_FILE, --snapshot-file SNAPSHOT_FILE
                                Set the snapshot filename for database dumps.
                                (default: None)
-am APPEND_MODE, --append-mode APPEND_MODE
                                The appendonly mode to be set. The avaiable options
                                are always, everysec, no. (default: None)
-af APPEND_FILE, --append-file APPEND_FILE
                                Set append only filename for merlin server container.
                                (default: None)
1.9. Merlin Server                                                                                      101
Merlin Documentation, Release 1.11.0
1.10 Celery
Merlin uses Celery, a Python based distributed task management system. Merlin uses Celery to queue work which is
processed by Celery workers.
Merlin queues tasks to the broker which receives and routes tasks. Merlin by default is configured to use RabbitMQ.
Celery has many functions, it defines the interface to the task broker, the backend results database and the workers that
will run the tasks.
The broker and backend are configured through the app.yaml file. A configuration for the rabbit ampq server is shown
below.
celery:
      # directory where Merlin looks for the following:
      # mysql-ca-cert.pem rabbit-client-cert.pem rabbit-client-key.pem redis.pass
      certs: /path/to/celery/config
broker:
      # can be rabbitmq, redis, rediss, or redis+sock
      name: rabbitmq
      #username: # defaults to your username unless changed here
      password: ~/.merlin/rabbit-password
      # server URL
      server: server.domain.com
      ### for rabbitmq connections ###
      #vhost: # defaults to your username unless changed here
      ### for redis+sock connections ###
      #socketname: the socket name your redis connection can be found on.
      #path: The path to the socket.
      ### for redis/rediss connections ###
      #port: The port number redis is listening on (default 6379)
      #db_num: The data base number to connect to.
      # ssl security
      #keyfile: /var/ssl/private/client-key.pem
      #certfile: /var/ssl/amqp-server-cert.pem
      #ca_certs: /var/ssl/myca.pem
      # This is optional and can be required, optional or none
      # (required is the default)
      #cert_reqs: required
results_backend:
      # Can be redis,rediss, mysql, db+ or memcached server
      # Only a few of these are directly configured by merlin
      name: redis
      dbname: dbname
      username: username
      # name of file where redis password is stored.
                                                                                                     (continues on next page)
102                                                                                    Chapter 1. Merlin Overview
                                                                            Merlin Documentation, Release 1.11.0
                                                                                                (continued from previous page)
       password: redis.pass
       server: server.domain.com
       # merlin will generate this key if it does not exist yet,
       # and will use it to encrypt all data over the wire to
       # your redis server.
       encryption_key: ~/.merlin/encrypt_data_key
       port: 6379
       db_num: 0
       # ssl security
       #keyfile: /var/ssl/private/client-key.pem
       #certfile: /var/ssl/amqp-server-cert.pem
       #ca_certs: /var/ssl/myca.pem
       # This is optional and can be required, optional or none
       # (required is the default)
       #cert_reqs: required
The default location for the app.yaml is in the merlin repo under the config directory. This default can be overridden by
files in one of two other locations. The current working directory is first checked for the app.yaml file, then the user’s
~/.merlin directory is checked.
The celery command needs application configuration for the specific module that includes celery, this is specified using
the -A <module> syntax. All celery commands should include the -A argument.
celery -A merlin
The merlin run command will define the tasks from the steps in the yaml file and then send them to the broker through
the celery broker interface. If these tasks are no longer needed or are incorrect, they can be purged by using one of
these commands:
celery -A merlin -Q <queue list> purge
# This is the equivalent of merlin purge <yaml file>
e.g.
celery -A merlin -Q merlin,queue2,queue3 purge
or with rabbitmq:
celery -A merlin amqp queue.purge <queue name>
e.g.
celery -A merlin amqp queue.purge merlin
a third option with rabbitmq is deleting the queue
celery -A merlin amqp queue.delete <queue>
e.g.
celery -A merlin amqp queue.delete merlin
1.10. Celery                                                                                                             103
Merlin Documentation, Release 1.11.0
1.10.1 Configuring celery workers
The common configurations used for the celery workers in the celery workers guide are not the best for HPC applica-
tions. Here are some parameters you may want to use for HPC specific workflows.
These options can be altered by setting the args for an entry of type worker in the merlin resources section.
The number of threads to use on each node of the HPC allocation is set through the --concurrency keyword. A good
choice for this is the number of simulations that can be run per node.
celery -A merlin worker --concurrency <num threads>
e.g.
# If the HPC simulation is a simple 1D short running sim
# then on Lassen you might want to use all Hardware threads.
celery -A merlin worker --concurrency 160
# If the HPC simulation will take the whole node you may want
# to limit this to only a few threads.
celery -A merlin worker --concurrency 2
The --prefetch-multiplier argument sets how many tasks are requested from the task server per worker thread.
If --concurrency is 2 and --prefetch-multiplier is 3, then 6 tasks will be requested from the task server by the
worker threads. Since HPC tasks are generally not short running tasks, the recommendation is to set this to 1.
celery -A merlin worker --prefetch-multiplier <num_tasks>
e.g.
celery -A merlin worker --prefetch-multiplier 1
The -O fair option is another parameter used for long running celery tasks. With this set, celery will only send tasks
to threads that are available to run them.
celery -A merlin worker -O fair
The -n option allows the workers to be given a unique name so multiple workers running tasks from different queues
may share the allocation resources. The names are automatically set to <queue name>.%h, where <queue name> is
from the task_queue config or merlin (default) and %h will resolve to the hostname of the compute node.
celery -A merlin worker -n <name>
e.g.
celery -A merlin worker -n merlin.%h
or
celery -A merlin worker -n queue_1.%h
On the toss3 nodes, the CPU affinity can be set for the worker processes. This is enabled by setting the environment
variable CELERY_AFFINIITY to the number of CPUs to skip. e.g. export CELERY_AFFINIITY=4 This will skip 4
CPUs between each celery worker thread.
104                                                                                   Chapter 1. Merlin Overview
                                                                          Merlin Documentation, Release 1.11.0
1.11 Virtual environments
This section provides a quick reference for using virtual environments for the Merlin project.
1.11.1 Creating a virtual environment
To create a new virtual environment:
$ python3 -m venv venv
  Caution: A virtual environment will need to be created for each system type. It’s recommended to name the
  virtual environment venv_<system> to make it easier to switch between them. This documentation will use venv
  for simplicity to reference the virtual environment.
Tip: Virtual environments provide an isolated environment for working on Python projects to avoid dependency
conflicts.
Activating a Virtualenv
Once the virtual environment is created it can be activated like so:
$ source venv/bin/activate
(venv) $
This will set the Python and Pip path to the virtual environment at venv/bin/python and venv/bin/pip respectively.
The virtual environment name should now display in the terminal, which means it is active. Any calls to pip will install
to the virtual environment.
Tip: To verify that Python and Pip are pointing to the virtual environment, run $ which python and $ which pip.
Deactivating a Virtualenv
Virtualenvs can be exited via the following:
(venv) $ deactivate
$
1.11. Virtual environments                                                                                         105
Merlin Documentation, Release 1.11.0
1.12 Spack
The virtualenv method is not the only method to install merlin in a separate python install. The spack method will
build python and all required modules for a specific set of configuration options. These options include the compiler
version, system type and python version. Merlin will then be installed in this specific version allowing for multiple
python versions on a single system without the need for a virtualenv. The py-merlin package builds with python3.6+.
1.12.1 Checkout spack
Get the latest version of spack from Github. This is independent from merlin so make sure merlin and spack are in
separate directories.
git clone https://github.com/spack/spack.git
# The merlin spack package is in the develop branch
git checkout develop
1.12.2 Setup spack
cd to spack directory
Source the setup-env.sh or setup-env.csh. This will put spack in your path and setup module access for later use.
This should be done every time the modules are used.
source ./share/spack/setup-env.sh
Add compilers if you haven’t already:
spack compiler add
To see the compilers.
spack compiler list
1.12.3 Build merlin
Build merlin, this will take a long time, be prepared to wait. It will build python and all python modules merlin needs
including numpy.
spack install py-merlin
The build will be done with the default compiler, in general this is the newest gcc compiler. You can choose a different
compiler by using the % syntax, this will create an entirely separate build and module.
spack install py-merlin%gcc@7.1.0
A different python version can be specified as part of the package config. To build merlin with python-3.6.8 you would
type:
spack install py-merlin^python@3.6.8
A tree of all of the packages and their dependencies needed to build the merlin package can be shown by using the spec
keyword.
106                                                                                    Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
spack spec py-merlin
1.12.4 Activate merlin
To use merlin you can activate the module.
spack activate py-merlin
or
spack activate py-merlin%gcc@7.1.0
or
spack activate py-merlin^python@3.6.8
1.12.5 Load python
The associated python module can then be loaded into your environment, this will only work if you have sourced the
setup-env.sh or setup-env.csh.
module avail python
example:
------ <path to>/spack/share/spack/modules/linux-rhel7-x86_64 -------
    python-3.6.8-gcc-8.1.0-4ilk3kn (L)
This will give you a list, the spack version will have a long hash associated with the name.
module load python-3.6.8-<compiler>-<hash>
e.g.
module load python-3.6.8-gcc-8.1.0-4ilk3kn
At this point the module specific python, merlin, maestro and celery will all be in your path.
1.13 Contributing
Welcome to the Merlin developer documentation! This section provides instructions for contributing to Merlin.
1.13.1 Getting Started
Follow the Getting Started documentation to setup your Merlin development environment.
Once your development is setup create a branch:
$ git checkout -b feature/<username>/description
Note: Other common types of branches besides feature are: bugfix, hotfix, or refactor.
1.13. Contributing                                                                                            107
Merlin Documentation, Release 1.11.0
Select the branch type that best represents the development.
Merlin follows a gitflow workflow. Updates to the develop branch are made via pull requests.
1.13.2 Developer Guide
This section provides Merlin’s guide for contributing features/bugfixes to Merlin.
1.13.3 Pull Request Checklist
 Warning: All pull requests must pass make tests prior to consideration!
To expedite review, please ensure that pull requests
    • Are from a meaningful branch name (e.g. feature/my_name/cool_thing)
    • Are being merged into the appropriate branch
    • Include testing for any new features
          – unit tests in tests/unit
          – integration tests in tests/integration
    • Include descriptions of the changes
          – a summary in the pull request
          – details in CHANGELOG.md
    • Ran make fix-style to adhere to style guidelines
    • Pass make tests; output included in pull request
    • Increment version number appropriately
          – in CHANGELOG.md
          – in merlin.__init__.py
    • Have squashed commits
1.13.4 Testing
All pull requests must pass unit and integration tests. To ensure that they do run
$ make tests
108                                                                                Chapter 1. Merlin Overview
                                                                                Merlin Documentation, Release 1.11.0
1.13.5 Python Code Style Guide
This section documents Merlin’s style guide. Unless otherwise specified, PEP-8 is the preferred coding style and PEP-
0257 for docstrings.
Note: make fix-style will automatically fix any style issues.
Merlin has style checkers configured. They can be run from the Makefile:
$ make check-style
1.13.6 Adding New Features to YAML Spec File
In order to conform to Maestro’s verification format introduced in Maestro v1.1.7, we now use json schema validation
to verify our spec file.
If you are adding a new feature to Merlin that requires a new block within the yaml spec file or a new property within
a block, then you are going to need to update the merlinspec.json file located in the merlin/spec/ directory. You also
may want to add additional verifications within the specification.py file located in the same directory.
Note: If you add custom verifications beyond the pattern checking that the json schema checks for, then you should
also add tests for this verification in the test_specification.py file located in the merlin/tests/unit/spec/ directory. Follow
the steps for adding new tests in the docstring of the TestCustomVerification class.
Adding a New Property
To add a new property to a block in the yaml file, you need to create a template for that property and place it in the
correct block in merlinspec.json. For example, say I wanted to add a new property called example that’s an integer
within the description block. I would modify the description block in the merlinspec.json file to look like this:
"DESCRIPTION": {
   "type": "object",
   "properties": {
      "name": {"type": "string", "minLength": 1},
      "description": {"type": "string", "minLength": 1},
      "example": {"type": "integer", "minimum": 1}
   },
   "required": ["name", "description"]
}
If you need help with json schema formatting, check out the step-by-step getting started guide.
That’s all that’s required of adding a new property. If you want to add your own custom verifications make sure to
create unit tests for them (see the note above for more info).
1.13. Contributing                                                                                                         109
Merlin Documentation, Release 1.11.0
Adding a New Block
Adding a new block is slightly more complicated than adding a new property. You will not only have to update the
merlinspec.json schema file but also add calls to verify that block within specification.py.
To add a block to the json schema, you will need to define the template for that entire block. For example, if I wanted
to create a block called country with two properties labeled name and population that are both required, it would
look like so:
"COUNTRY": {
   "type": "object",
   "properties": {
      "name": {"type": "string", "minLength": 1},
      "population": {
        "anyOf": [
           {"type": "string", "minLength": 1},
           {"type": "integer", "minimum": 1}
        ]
      }
   },
   "required": ["name", "capital"]
}
Here, name can only be a string but population can be both a string and an integer. For help with json schema
formatting, check out the step-by-step getting started guide.
The next step is to enable this block in the schema validation process. To do this we need to:
    1. Create a new method called verify_<your_block_name>() within the MerlinSpec class
    2. Call the YAMLSpecification.validate_schema() method provided to us via Maestro in your new method
    3. Add a call to verify_<your_block_name>() inside the verify() method
If you add your own custom verifications on top of this, please add unit tests for them.
1.14 Docker
Merlin has a simple Dockerfile description for running a container with all requirements installed.
1.14.1 Build the container
The docker container can be built by building in the top level merlin directory.
docker build -t merlin .
This will create a merlin:latest image in your docker image collection with a user “merlinu” and a WORKDIR set to
/home/merlinu.
docker images
110                                                                                    Chapter 1. Merlin Overview
                                                                           Merlin Documentation, Release 1.11.0
1.14.2 Run the container
The container can be run in detached mode to provide both the merlin and celery commands
docker run --rm -td --name my-merlin merlin
alias merlin="docker exec my-merlin merlin"
alias celery="docker exec my-merlin celery"
Examples can be run through docker containers by first starting a server for the broker and backend. The server can be
a redis or rabbitmq , for this demonstration a redis server will be used. The backend will always be a redis server.
docker pull redis
docker run -d -p 6379:6379 --name my-redis redis
A local output directory can be defined by using the --volume docker arguments. It is recommended that a fixed
directory be used for the --volume argument. The merlin docker container is linked to the redis server above by using
the --link option.
# Create local working directory
mkdir $HOME/merlinu
cd $HOME/merlinu
docker pull llnl/merlin
docker run --rm -td --name my-merlin --link my-redis --volume "$HOME/merlinu":/home/
 ˓→merlinu llnl/merlin
alias merlin="docker exec my-merlin merlin"
alias celery="docker exec my-merlin celery"
# Create the $HOME/merlinu/.merlin/app.yaml using redis
merlin config --broker redis
<edit $HOME/merlinu/.merlin/app.yaml and change the broker and backend server: variables␣
 ˓→to    my-redis>
# Copy an example to the local dir
merlin example feature_demo
# Run a test run without workers
merlin run feature_demo/feature_demo.yaml --dry --local
# Define the tasks and load them on the broker
merlin run feature_demo/feature_demo.yaml
# Start workers to pull tasks from the server and run them in the container
merlin run-workers feature_demo/feature_demo.yaml
A shell can started in the container by using the --entrypoint command. If the user would like to examine the
container contents, they can use a shell as the entry point.
docker run --rm -ti --volume "$HOME/merlinu":/home/merlinu --entrypoint="/bin/bash"␣
 ˓→merlin
1.14. Docker                                                                                                         111
