<commit_before>---
layout: page
permalink: /profiling/index.html
title: Performance Profiling and Tuning
---

After developing a functional Legion application, it is usually
necessary to profile and tune for performance. This page covers
techniques to achieving high performance in Legion applications.

  * [Preliminary Steps](#preliminary-steps)
  * [Legion Prof](#legion-prof)
  * [General Optimization Techniques](#general-optimization-techniques)
  * Legion Configuration:
      * [Machine Configuration](#machine-configuration)
      * [Legion Runtime Performance Flags](#legion-runtime-performance-flags)

## Preliminary Steps

When performance is desired, *always* build Legion in release mode
(i.e. `DEBUG=0 make`). Release mode ensures that expensive assertions
and checks are disabled, and that appropriate optimization flags are
used in the build. Also, remember to remove any unnecessary build
flags (e.g. `-DPRIVILEGE_CHECKS`, `-DBOUNDS_CHECKS`, or
`-DLEGION_SPY`).

Set `OUTPUT_LEVEL` at an appropriate level. The default
(`LEVEL_DEBUG`) is usually acceptable as dynamic checks are sufficient
to ensure that the extra logging statements are not a performance
problem. However, a higher level such as `LEVEL_SPEW` might cause
problems, even when disabled on the command-line.

If using GASNet, ensure that [GASNet has been properly configured and
optimized](/gasnet/). GASNet can be important for performance on
multi-node runs.

## Legion Prof

Before taking steps towards optimizing a program, it is usually
helpful to profile the application in question. Legion has a
task-level profiler called Legion Prof which is used for this purpose.

Legion Prof is compiled into all builds by default, so no additional
compile-time flags are required. However, it is always recommended to
compile Legion in release mode (i.e. `DEBUG=0`) before measuring
performance.

To profile an application, run with `-hl:prof <N>` where `N` is the
number of nodes to be profiled. (`N` can be less than the total number
of nodes---this profiles a subset of the nodes.) Use the `-logfile`
flag to save the output from each node to a separate file (see
[logging documentation](/debugging/#logging-infrastructure)).
Finally, pass the resulting log files to `legion_prof.py`.

{% highlight bash %}
DEBUG=0 make
./app -hl:prof <N> -logfile prof_%.log
$LG_RT_DIR/tools/legion_prof.py prof_*.log
{% endhighlight %}

This will generate a subdirectory called `legion_prof` under the
current directory, including a file named `index.html`. Open this file
in your browser.

The web pages generated by Legion Prof are interactive. To zoom in,
click and drag the mouse over a region of interest. For other options,
use the `?` to show a help message.

A sample of Legion Prof's output is shown below.

![](/images/legion_prof.jpg)

## General Optimization Techniques

In general there are three primary factors that contribute
to the performance of a Legion application.

 1. Overall task and data movement throughput
 2. Latency of the critical path through the task/copy graph
 3. Runtime overhead

There are many variables which can impact each of these
three performance metrics including the decisions made by the
application mapper(s), the size and shape of the 
region trees, the selected regions and fields requested
by tasks, and the settings of many of the variables
specified on this page. The goal of performance profiling
is to determine which variables are impacting performance
and tune them for the best performance. Due to the large
parameter spaces in Legion programs this is a difficult
problem to solve in general. We therefore give some basic
suggestions that we use when tuning Legion applications.

When tuning our Legion applications we usually make heavy<commit_msg>Fix typo.<commit_after>---
layout: page
permalink: /profiling/index.html
title: Performance Profiling and Tuning
---

After developing a functional Legion application, it is usually
necessary to profile and tune for performance. This page covers
techniques to achieving high performance in Legion applications.

  * [Preliminary Steps](#preliminary-steps)
  * [Legion Prof](#legion-prof)
  * [General Optimization Techniques](#general-optimization-techniques)
  * Legion Configuration:
      * [Machine Configuration](#machine-configuration)
      * [Legion Runtime Performance Flags](#legion-runtime-performance-flags)

## Preliminary Steps

When performance is desired, *always* build Legion in release mode
(i.e. `DEBUG=0 make`). Release mode ensures that expensive assertions
and checks are disabled, and that appropriate optimization flags are
used in the build. Also, remember to remove any unnecessary build
flags (e.g. `-DPRIVILEGE_CHECKS`, `-DBOUNDS_CHECKS`, or
`-DLEGION_SPY`).

Set `OUTPUT_LEVEL` at an appropriate level. The default
(`LEVEL_DEBUG`) is usually acceptable as dynamic checks are sufficient
to ensure that the extra logging statements are not a performance
problem. However, a higher level such as `LEVEL_SPEW` might cause
problems, even when disabled on the command-line.

If using GASNet, ensure that [GASNet has been properly configured and
optimized](/gasnet/). GASNet can be important for performance on
multi-node runs.

## Legion Prof

Before taking steps towards optimizing a program, it is usually
helpful to profile the application in question. Legion has a
task-level profiler called Legion Prof which is used for this purpose.

Legion Prof is compiled into all builds by default, so no additional
compile-time flags are required. However, it is always recommended to
compile Legion in release mode (i.e. `DEBUG=0`) before measuring
performance.

To profile an application, run with `-hl:prof <N>` where `N` is the
number of nodes to be profiled. (`N` can be less than the total number
of nodes---this profiles a subset of the nodes.) Use the `-logfile`
flag to save the output from each node to a separate file (see
[logging documentation](/debugging/#logging-infrastructure)).
Finally, pass the resulting log files to `legion_prof.py`.

{% highlight bash %}
DEBUG=0 make
./app -hl:prof <N> -logfile prof_%.log
$LG_RT_DIR/tools/legion_prof.py prof_*.log
{% endhighlight %}

This will generate a subdirectory called `legion_prof` under the
current directory, including a file named `index.html`. Open this file
in your browser.

The web pages generated by Legion Prof are interactive. To zoom in,
click and drag the mouse over a region of interest. For other options,
use the `?` key to show a help message.

A sample of Legion Prof's output is shown below.

![](/images/legion_prof.jpg)

## General Optimization Techniques

In general there are three primary factors that contribute
to the performance of a Legion application.

 1. Overall task and data movement throughput
 2. Latency of the critical path through the task/copy graph
 3. Runtime overhead

There are many variables which can impact each of these
three performance metrics including the decisions made by the
application mapper(s), the size and shape of the 
region trees, the selected regions and fields requested
by tasks, and the settings of many of the variables
specified on this page. The goal of performance profiling
is to determine which variables are impacting performance
and tune them for the best performance. Due to the large
parameter spaces in Legion programs this is a difficult
problem to solve in general. We therefore give some basic
suggestions that we use when tuning Legion applications.

When tuning our Legion applications we usually make heavy
{'fraction_non_alphanumeric': 0.06243372216330859, 'fraction_numerical': 0.0015906680805938495, 'mean_word_length': 4.698640483383686, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '563002', 'n_tokens_mistral': 1976, 'n_tokens_neox': 1865, 'n_words': 1031}