<commit_before>Parallel execution
------------------

The pipeline runs in parallel in two different ways:

-  multiple cores -- Analyses will run in parallel using multiple cores
   on a single machine. This requires only the ``multiprocessing``
   Python library, included by default with most Python installations.

-  parallel messaging -- This allows scaling beyond the cores
   available on a single machine, and requires multiple machines
   with a shared filesystem like standard cluster environments.
   Machine to machine communication occurs via messaging, using the
   `IPython parallel`_ framework.

Multiple cores
~~~~~~~~~~~~~~
Running using multiple cores only requires setting the ``-n``
command line flag::

    bcbio_nextgen.py bcbio_sample.yaml -t local -n 12

IPython parallel
~~~~~~~~~~~~~~~~

`IPython parallel`_ provides a distributed framework for performing
parallel computation in standard cluster environments. The
bcbio-nextgen setup script installs both IPython and `pyzmq`_, which
provides Python bindings for the `ZeroMQ`_ messaging library. The only
additional requirement is that the work directory where you run the
analysis is accessible to all processing nodes. This is typically
accomplished with a distributed file system like
`NFS`_, `Gluster`_ or `Lustre`_.

Run an analysis using ipython for parallel execution::

    bcbio_nextgen.py bcbio_sample.yaml -t ipython -n 12 -s lsf -q queue

The ``-s`` flag specifies a type of scheduler to use ``(lsf, sge, torque)``.

The ``-q`` flag specifies the queue to submit jobs to.

The ``-n`` flag defines the total number of cores to use on the
cluster during processing. The framework will select the appropriate
number of cores and type of cluster (single core versus multi-core) to
use based on the pipeline stage (see the :ref:`internals-parallel`
section in the internals documentation for more details). For
multiple core steps, the number of cores to use for programs like
``bwa``, ``novoalign`` and ``gatk`` comes from the
:ref:`config-resources` section of the configuration.
Ensure the ``cores`` specification matches the physical cores
available on machines in your cluster, and the pipeline will divide
the total cores specified by ``-n`` into the appropriate number of
multicore jobs to run.

The pipeline default parameters assume a system with minimal time to
obtain processing cores and consistent file system accessibility. These
defaults allow the system to fail fast in the case of cluster issues
which need diagnosis. For running on shared systems with high resource
usage and potential failures due to intermittent cluster issues, there
are turning parameters that increase resiliency. The ``--timeout``
flag specifies the numbers of minutes to wait for a cluster to start
up before timing out. This defaults to 15 minutes. The ``--retries``
flag specify the number of times to retry a job on failure. In systems
with transient distributed file system hiccups like lock errors or disk
availability, this will provide recoverability at the cost of
resubmitting jobs that may have failed for reproducible reasons.

Finally, the ``-r resources`` flag specifies resource options to pass along
to the underlying queue scheduler. This currently supports SGE's
``-l`` parameter, Torque's ``-l`` parameter, LSF and SLURM native flags. This allows specification
or resources to the scheduler (see the `qsub man page`_). You may specify multiple
resources, so ``-r mem=4g -r ct=01:40:00``
translates to ``-l mem=4g -l ct=01:40:00`` when passed to ``qsub`` or
``-r "account=a2010002;timelimit=04:00:00"`` when using SLURM, for
instance. SLURM and Torque support specification of an account parameter with
``-r account=your_name``, which IPython transfers into ``-A``.

SGE supports special parameters passed using resources to help handle the
heterogeneity of possible setups. Specify the `SGE parallel environment`_ to use
for submitting multicore jobs with ``-r pename=your_pe``. Since this setup is
system specific it is hard to write general code for find a suitable
environment. Specifically, when there are multiple usable parallel environments,
it will select the first one which may not be correct. Manually specifying it
with a ``pename=`` flag to resources will ensure correct selection of the right
environment. To specify an advanced reservation with the ``-ar`` flag, use
``-r ar=ar_id``. To specify an alternative memory management model instead of
``mem_free`` use ``-r memtype=approach``. It is further recommended to configure
``mem_free`` (or any other chosen memory management model) as a consumable, requestable
resource in SGE to prevent overfilling hosts that do not have sufficient memory per slot. 
This can be done in two steps. First, launch ``qmon`` as an admin,
select ``Complex Configuration`` in qmon, click on ``mem_free`, 
under the ``Consumable`` dialog select ``JOB`` (instead of ``YES`` or ``NO``) and
finally click ``Modify`` for the changes to take effect. Secondly, for each host in
the queue, configure ``mem_free`` as a complex value. If a host called ``myngshost`` 
has 128GB of RAM, the corresponding command would be 
``qconf -mattr exechost complex_values mem_free=128G myngshost``

There are also special ``-r`` resources parameters to support the pipeline:

- ``-r conmem=4`` -- Specify the memory for the controller process, in Gb. This
  currently applies to SLURM processing and defaults to 4Gb.

- ``-r mincores=16`` -- Specify the minimum number of cores to batch together
  for parallel single core processes like variant calling. This will run
  multiple processes together under a single submission to allow sharing of
  resources like memory, which is helpful when a small percentage of the time a
  process like variant calling will use a lot of memory. By default, bcbio will
  calculate ``mincores`` based on specifications for multicore calling so this
  doesn't normally require a user to set.

.. _qsub man page: http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html
.. _IPython parallel: http://ipython.org/ipython-doc/dev/index.html
.. _pyzmq: https://github.com/zeromq/pyzmq
.. _ZeroMQ: http://www.zeromq.org/
.. _Gluster: http://www.gluster.org/
.. _Lustre: http://wiki.lustre.org/index.php/Main_Page
.. _NFS: https://en.wikipedia.org/wiki/Network_File_System_%28protocol%29
.. _SGE parallel environment: https://blogs.oracle.com/templedf/entry/configuring_a_new_parallel_environment

Troubleshooting
===============<commit_msg>Ensure resources examples do not use deprecated semi-colon syntax. Fixes #772<commit_after>Parallel execution
------------------

The pipeline runs in parallel in two different ways:

-  multiple cores -- Analyses will run in parallel using multiple cores
   on a single machine. This requires only the ``multiprocessing``
   Python library, included by default with most Python installations.

-  parallel messaging -- This allows scaling beyond the cores
   available on a single machine, and requires multiple machines
   with a shared filesystem like standard cluster environments.
   Machine to machine communication occurs via messaging, using the
   `IPython parallel`_ framework.

Multiple cores
~~~~~~~~~~~~~~
Running using multiple cores only requires setting the ``-n``
command line flag::

    bcbio_nextgen.py bcbio_sample.yaml -t local -n 12

IPython parallel
~~~~~~~~~~~~~~~~

`IPython parallel`_ provides a distributed framework for performing
parallel computation in standard cluster environments. The
bcbio-nextgen setup script installs both IPython and `pyzmq`_, which
provides Python bindings for the `ZeroMQ`_ messaging library. The only
additional requirement is that the work directory where you run the
analysis is accessible to all processing nodes. This is typically
accomplished with a distributed file system like
`NFS`_, `Gluster`_ or `Lustre`_.

Run an analysis using ipython for parallel execution::

    bcbio_nextgen.py bcbio_sample.yaml -t ipython -n 12 -s lsf -q queue

The ``-s`` flag specifies a type of scheduler to use ``(lsf, sge, torque)``.

The ``-q`` flag specifies the queue to submit jobs to.

The ``-n`` flag defines the total number of cores to use on the
cluster during processing. The framework will select the appropriate
number of cores and type of cluster (single core versus multi-core) to
use based on the pipeline stage (see the :ref:`internals-parallel`
section in the internals documentation for more details). For
multiple core steps, the number of cores to use for programs like
``bwa``, ``novoalign`` and ``gatk`` comes from the
:ref:`config-resources` section of the configuration.
Ensure the ``cores`` specification matches the physical cores
available on machines in your cluster, and the pipeline will divide
the total cores specified by ``-n`` into the appropriate number of
multicore jobs to run.

The pipeline default parameters assume a system with minimal time to
obtain processing cores and consistent file system accessibility. These
defaults allow the system to fail fast in the case of cluster issues
which need diagnosis. For running on shared systems with high resource
usage and potential failures due to intermittent cluster issues, there
are turning parameters that increase resiliency. The ``--timeout``
flag specifies the numbers of minutes to wait for a cluster to start
up before timing out. This defaults to 15 minutes. The ``--retries``
flag specify the number of times to retry a job on failure. In systems
with transient distributed file system hiccups like lock errors or disk
availability, this will provide recoverability at the cost of
resubmitting jobs that may have failed for reproducible reasons.

Finally, the ``-r resources`` flag specifies resource options to pass along
to the underlying queue scheduler. This currently supports SGE's
``-l`` parameter, Torque's ``-l`` parameter, LSF and SLURM native flags. This allows specification
or resources to the scheduler (see the `qsub man page`_). You may specify multiple
resources, so ``-r mem=4g -r ct=01:40:00``
translates to ``-l mem=4g -l ct=01:40:00`` when passed to ``qsub`` or
``-r "account=a2010002" -r "timelimit=04:00:00"`` when using SLURM, for
instance. SLURM and Torque support specification of an account parameter with
``-r account=your_name``, which IPython transfers into ``-A``.

SGE supports special parameters passed using resources to help handle the
heterogeneity of possible setups. Specify the `SGE parallel environment`_ to use
for submitting multicore jobs with ``-r pename=your_pe``. Since this setup is
system specific it is hard to write general code for find a suitable
environment. Specifically, when there are multiple usable parallel environments,
it will select the first one which may not be correct. Manually specifying it
with a ``pename=`` flag to resources will ensure correct selection of the right
environment. To specify an advanced reservation with the ``-ar`` flag, use
``-r ar=ar_id``. To specify an alternative memory management model instead of
``mem_free`` use ``-r memtype=approach``. It is further recommended to configure
``mem_free`` (or any other chosen memory management model) as a consumable, requestable
resource in SGE to prevent overfilling hosts that do not have sufficient memory per slot.
This can be done in two steps. First, launch ``qmon`` as an admin,
select ``Complex Configuration`` in qmon, click on ``mem_free`,
under the ``Consumable`` dialog select ``JOB`` (instead of ``YES`` or ``NO``) and
finally click ``Modify`` for the changes to take effect. Secondly, for each host in
the queue, configure ``mem_free`` as a complex value. If a host called ``myngshost``
has 128GB of RAM, the corresponding command would be
``qconf -mattr exechost complex_values mem_free=128G myngshost``

There are also special ``-r`` resources parameters to support pipeline configuration:

- ``-r conmem=4`` -- Specify the memory for the controller process, in Gb. This
  currently applies to SLURM processing and defaults to 4Gb.

- ``-r mincores=16`` -- Specify the minimum number of cores to batch together
  for parallel single core processes like variant calling. This will run
  multiple processes together under a single submission to allow sharing of
  resources like memory, which is helpful when a small percentage of the time a
  process like variant calling will use a lot of memory. By default, bcbio will
  calculate ``mincores`` based on specifications for multicore calling so this
  doesn't normally require a user to set.

.. _qsub man page: http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html
.. _IPython parallel: http://ipython.org/ipython-doc/dev/index.html
.. _pyzmq: https://github.com/zeromq/pyzmq
.. _ZeroMQ: http://www.zeromq.org/
.. _Gluster: http://www.gluster.org/
.. _Lustre: http://wiki.lustre.org/index.php/Main_Page
.. _NFS: https://en.wikipedia.org/wiki/Network_File_System_%28protocol%29
.. _SGE parallel environment: https://blogs.oracle.com/templedf/entry/configuring_a_new_parallel_environment

Troubleshooting
===============
{'fraction_non_alphanumeric': 0.08044393417527745, 'fraction_numerical': 0.007577497129735935, 'mean_word_length': 4.675933970460469, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2820191', 'n_tokens_mistral': 3557, 'n_tokens_neox': 3356, 'n_words': 1772}