<commit_before>Tips for running large risk calculations
========================================

Scenario risk calculations usually do not pose a performance problem,
since they involve a single rupture and a limited geography for analysis. 
Some event-based risk calculations, however, may involve millions of ruptures
and exposures spanning entire countries or even larger regions. This section
offers some practical tips for running large event based risk calculations, 
especially ones involving large logic trees, and proposes techniques that might
be used to make an intractable calculation tractable.

Understanding the hazard
------------------------

Event-based calculations are typically dominated by the hazard component
(unless there are lots of assets aggregated on few hazard sites) and
therefore the first thing to do is to estimate the size of the hazard,
i.e. the number of GMFs that will be produced. Since we are talking about
a large calculation, first of all we need reduce it to a size that is 
guaranteed to run quickly. The simplest way to do that is to reduce the 
parameters directly affecting the number of ruptures generated, i.e.

- investigation_time
- ses_per_logic_tree_path
- number_of_logic_tree_samples

For instance, if you have ``ses_per_logic_tree_path = 10,000`` reduce
it to 10, run the calculation and you will see in the log something
like this::

  [2018-12-16 09:09:57,689 #35263 INFO] Received
  {'gmfdata': '752.18 MB', 'hcurves': '224 B', 'indices': '29.42 MB'}

The amount of GMFs generated for the reduced calculation is 752.18 MB; 
and since the calculation has been reduced by a factor of 1,000, 
the full computation is likely to generate around 750 GB of GMFs. 
Even if you have sufficient disk space to store this large quantity of GMFs, 
most likely you will run out of memory. Even if the hazard part of the 
calculation manages to run to completion, the risk part of the calculation
is very likely to fail — managing 750 GB of GMFs is beyond the current 
capabilities of the engine. Thus, you will have to find ways to reduce the
size of the computation. 

A good start would be to carefully set the parameters 
``minimum_magnitude`` and ``minimum_intensity``:

- ``minimum_magnitude`` is a scalar or a dictionary keyed by tectonic region;
  the engine will discard ruptures with magnitudes below the given threshoulds
- ``minimum_intensity`` is a scalar or a dictionary keyed by the intensity
  measure type; the engine will discard GMFs below the given intensity threshoulds

Choosing reasonable cutoff thresholds with these parameters can significantly
reduce the size of your computation when there are a large number of 
small magnitude ruptures or low intensity GMFs being generated, which may have
a negligible impact on the damage or losses, and thus could be safely discarded.

region_grid_spacing
---------------------

In our experience, the most common error made by out users is to
compute the hazard at the sites of the exposure. The issue is that it
very possible to have exposures with millions of assets on millions of
distinct hazard sites. Computing the GMFs for millions of sites is
hard or even impossible (there is a limit of 4 billion rows on the
size of the GMF table in the datastore).  Even in the cases when
computing the hazard is possible, then computing the risk starting
from an extremely large amount of GMFs will likely be impossible, due
to memory/runtime constraints.

The second most common error is to use an extremely fine grid for the
site model. Remember that if you have a resolution of 250 meters, a
square of 250 km x 250 km will contain one million sites, which is
definitely too much. The engine when designed when the site models
had resolutions around 5-10 km, i.e. of the same order of the hazard
grid, while nowadays the vs30 fields have a much larger resolution.

Both problems can be solved in a simple way by specifying the
``region_grid_spacing`` parameter. Make it large enough that the
resulting number of sites becomes reasonable and you are done.
You will loose some precision, but that is preferable to not
being able to run the calculation. You will need to run a sensitivity
analysis with different values of ``region_grid_spacing`` parameter
to make sure that you get consistent results, but that's it.

Once a ``region_grid_spacing`` is specified, the engine computes the
convex hull of the exposure sites and builds a grid of hazard sites,
associating the site parameters from the closest site in the site model
and discarding sites in region where there are no assets (i.e. more
distant than ``region_grid_spacing * sqrt(2)``). The precise logic
is encoded in the function
``openquake.commonlib.readinput.get_sitecol_assetcol``, if you want
to know the nitty-gritty details.

Our recommendation is to use the command ``oq prepare_site_model`` to
apply such logic before starting a calculation and thus producing a
custom site model file tailored to your exposure (see the section
:ref:`prepare_site_model`).


Collapsing of branches
----------------------

When one is not interested in the uncertainty around the loss
estimates and cares more about the mean estimates, all of the
source model branches can be "collapsed" into one branch. Using the
collapsed source model should yield the same mean hazard or loss
estimates as using the full source model logic tree and then computing
the weighted mean of the individual branch results.

Similarly, the GMPE logic tree for each tectonic region can also be "collapsed"
into a single branch. Using a single collapsed GMPE for each TRT
should also yield the same mean hazard estimates as using the full
GMPE logic tree and then computing the weighted mean of the individual
branch results. This has become possible through the introduction of 
`AvgGMPE feature <https://github.com/gem/oq-engine/blob/engine-3.9/openquake/qa_tests_data/classical/case_19/gmpe_logic_tree.xml#L26-L40>`_ in version 3.9.

Using ``collect_rlzs=true``
----------------------------

Since version 3.12 the engine recognizes a flag ``collect_rlzs`` in
the risk configuration file, which by default is false. When the flag
is set to true, then the hazard realizations are collected together
when computing the risk results and considered as one. This is
possible only when the weights of the realizations are all equal,
otherwise the engine raises an error. Collecting the realizations
makes the calculation of the losses and loss curves much faster and
more memory efficient. It is the recommended way to proceed when you
are interested only in mean results.

Splitting the calculation into subregions
-----------------------------------------

If one is interested in propagating the full uncertainty in the source
models or ground motion models to the hazard or loss estimates,
collapsing the logic trees into a single branch to reduce
computational expense is not an option. But before going through the
effort of trimming the logic trees, there is an interim step that must
be explored, at least for large regions like the entire continental United States.
This step is to geographically divide the large region into logical smaller
subregions, such that the contribution to the hazard or losses in one
subregion from the other subregions is negligibly small or even zero. 
The effective realizations in each of the subregions will then be much 
fewer than when trying to cover the entire large region in a single
calculation.


Trimming of the logic-trees or sampling of the branches
-------------------------------------------------------

Trimming or sampling may be necessary if the following two
conditions hold:

1. You are interested in propagating the full uncertainty to the<commit_msg>[ci skip]<commit_after>Tips for running large risk calculations
========================================

Scenario risk calculations usually do not pose a performance problem,
since they involve a single rupture and a limited geography for analysis. 
Some event-based risk calculations, however, may involve millions of ruptures
and exposures spanning entire countries or even larger regions. This section
offers some practical tips for running large event based risk calculations, 
especially ones involving large logic trees, and proposes techniques that might
be used to make an intractable calculation tractable.

Understanding the hazard
------------------------

Event-based calculations are typically dominated by the hazard component
(unless there are lots of assets aggregated on few hazard sites) and
therefore the first thing to do is to estimate the size of the hazard,
i.e. the number of GMFs that will be produced. Since we are talking about
a large calculation, first of all we need reduce it to a size that is 
guaranteed to run quickly. The simplest way to do that is to reduce the 
parameters directly affecting the number of ruptures generated, i.e.

- investigation_time
- ses_per_logic_tree_path
- number_of_logic_tree_samples

For instance, if you have ``ses_per_logic_tree_path = 10,000`` reduce
it to 10, run the calculation and you will see in the log something
like this::

  [2018-12-16 09:09:57,689 #35263 INFO] Received
  {'gmfdata': '752.18 MB', 'hcurves': '224 B', 'indices': '29.42 MB'}

The amount of GMFs generated for the reduced calculation is 752.18 MB; 
and since the calculation has been reduced by a factor of 1,000, 
the full computation is likely to generate around 750 GB of GMFs. 
Even if you have sufficient disk space to store this large quantity of GMFs, 
most likely you will run out of memory. Even if the hazard part of the 
calculation manages to run to completion, the risk part of the calculation
is very likely to fail — managing 750 GB of GMFs is beyond the current 
capabilities of the engine. Thus, you will have to find ways to reduce the
size of the computation. 

A good start would be to carefully set the parameters 
``minimum_magnitude`` and ``minimum_intensity``:

- ``minimum_magnitude`` is a scalar or a dictionary keyed by tectonic region;
  the engine will discard ruptures with magnitudes below the given threshoulds
- ``minimum_intensity`` is a scalar or a dictionary keyed by the intensity
  measure type; the engine will discard GMFs below the given intensity threshoulds

Choosing reasonable cutoff thresholds with these parameters can significantly
reduce the size of your computation when there are a large number of 
small magnitude ruptures or low intensity GMFs being generated, which may have
a negligible impact on the damage or losses, and thus could be safely discarded.

region_grid_spacing
---------------------

In our experience, the most common error made by out users is to
compute the hazard at the sites of the exposure. The issue is that it
very possible to have exposures with millions of assets on millions of
distinct hazard sites. Computing the GMFs for millions of sites is
hard or even impossible (there is a limit of 4 billion rows on the
size of the GMF table in the datastore).  Even in the cases when
computing the hazard is possible, then computing the risk starting
from an extremely large amount of GMFs will likely be impossible, due
to memory/runtime constraints.

The second most common error is to use an extremely fine grid for the
site model. Remember that if you have a resolution of 250 meters, a
square of 250 km x 250 km will contain one million sites, which is
definitely too much. The engine when designed when the site models
had resolutions around 5-10 km, i.e. of the same order of the hazard
grid, while nowadays the vs30 fields have a much larger resolution.

Both problems can be solved in a simple way by specifying the
``region_grid_spacing`` parameter. Make it large enough that the
resulting number of sites becomes reasonable and you are done.
You will loose some precision, but that is preferable to not
being able to run the calculation. You will need to run a sensitivity
analysis with different values of ``region_grid_spacing`` parameter
to make sure that you get consistent results, but that's it.

Once a ``region_grid_spacing`` is specified, the engine computes the
convex hull of the exposure sites and builds a grid of hazard sites,
associating the site parameters from the closest site in the site model
and discarding sites in region where there are no assets (i.e. more
distant than ``region_grid_spacing * sqrt(2)``). The precise logic
is encoded in the function
``openquake.commonlib.readinput.get_sitecol_assetcol``, if you want
to know the nitty-gritty details.

Our recommendation is to use the command ``oq prepare_site_model`` to
apply such logic before starting a calculation and thus producing a
custom site model file tailored to your exposure (see the section
:ref:`prepare_site_model`).


Collapsing of branches
----------------------

When one is not interested in the uncertainty around the loss
estimates and cares more about the mean estimates, all of the
source model branches can be "collapsed" into one branch. Using the
collapsed source model should yield the same mean hazard or loss
estimates as using the full source model logic tree and then computing
the weighted mean of the individual branch results.

Similarly, the GMPE logic tree for each tectonic region can also be "collapsed"
into a single branch. Using a single collapsed GMPE for each TRT
should also yield the same mean hazard estimates as using the full
GMPE logic tree and then computing the weighted mean of the individual
branch results. This has become possible through the introduction of 
`AvgGMPE feature <https://github.com/gem/oq-engine/blob/engine-3.9/openquake/qa_tests_data/classical/case_19/gmpe_logic_tree.xml#L26-L40>`_ in version 3.9.

Using ``collect_rlzs=true``
----------------------------

Since version 3.12 the engine recognizes a flag ``collect_rlzs`` in
the risk configuration file, which by default is false. When the flag
is set to true, then the hazard realizations are collected together
when computing the risk results and considered as one. This is
possible only when the weights of the realizations are all equal,
otherwise the engine raises an error. Collecting the realizations
makes the calculation of the losses and loss curves much faster and
more memory efficient. It is the recommended way to proceed when you
are interested only in mean results.

Splitting the calculation into subregions
-----------------------------------------

If one is interested in propagating the full uncertainty in the source
models or ground motion models to the hazard or loss estimates,
collapsing the logic trees into a single branch to reduce
computational expense is not an option. But before going through the
effort of trimming the logic trees, there is an interim step that must
be explored, at least for large regions like the entire continental United States.
This step is to geographically divide the large region into logical smaller
subregions, such that the contribution to the hazard or losses in one
subregion from the other subregions is negligibly small or even zero. 
The effective realizations in each of the subregions will then be much 
fewer than when trying to cover the entire large region in a single
calculation.

NB: ``collect_rlzs`` has no effect at all on the hazard results.

Trimming of the logic-trees or sampling of the branches
-------------------------------------------------------

Trimming or sampling may be necessary if the following two
conditions hold:

1. You are interested in propagating the full uncertainty to the
{'fraction_non_alphanumeric': 0.061097176667309794, 'fraction_numerical': 0.0110618046176603, 'mean_word_length': 4.108409986859396, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '735307', 'n_tokens_mistral': 3942, 'n_tokens_neox': 3598, 'n_words': 2357}