<commit_before>===
LWR
===

This project is a Python server application that allows a 
`Galaxy <http://galaxyproject.org>`_ server to run jobs on remote 
systems (including Windows) without requiring a shared mounted 
file systems. Input files, scripts, and
config files are transferred to the remote system, the job is
executed, and the results are transferred back to the Galaxy server.

------------------
Configuring Galaxy
------------------

A Galaxy tool can be configured to be executed remotely via LWR by
adding a line to the ``universe.ini`` file under the ``galaxy:tool_runners``
section with the format::

    <tool_id> = lwr://http://<lwr_host>:<lwr_port>

As an example, if a host named remotehost is running the LWR server
application on port ``8913``, then the tool with id ``test_tool`` can be
configured to run remotely on remotehost by adding the following line
to ``universe.ini``::

    test_tool = lwr://http://remotehost:8913

Remember this must be added after the ``[galaxy:tool_runners]`` header in
the ``universe.ini`` file.

---------------
Downloading LWR
---------------

The LWR server application is distributed as a Python project and can
be obtained via mercurial from bitbucket.org using the following
command::

    hg clone http://bitbucket.org/jmchilton/lwr

----------------
LWR Dependencies
----------------

Several Python packages must be installed to run the LWR server. These
can either be installed into a Python ``virtualenv`` or into your
system wide Python environment using ``easy_install``. Instructions
for both are outlined below:

virtualenv
----------

The script ``setup_venv.sh`` distributed with the LWR server is a
short-cut for \*nix machines to setup a Python environment (including
the installation of virtualenv). Full details for installation
suitable for \*nix or Windows are as follows.

1. Install `virtualenv <http://www.virtualenv.org/en/latest/#installation>`_::

    pip install virtualenv

2. Create a new Python environment::

    virtualenv -q --no-site-packages .venv

3. Activate environment (varies by OS). 

From a Linux or MacOS terminal::

    . .venv/bin/activate

From a Windows terminal::

    .venv\Scripts\activate

4. Install required dependencies into this virtual environment::

    pip install -r tools/pip-requires

easy_install
------------

Install python setuptools for your platform, more details on how to do
this can be found `here <http://pypi.python.org/pypi/setuptools>`_.

The ``easy_install`` command line application will be installed as part of
setuptools. Use the following command to install the needed packages
via ``easy_install``::

    easy_install paste wsgiutils PasteScript PasteDeploy simplejson webob pyOpenSSL

``pyOpenSSL`` is only required if LWR is configured to use HTTPS/SSL.

----------------------------------
Running the LWR Server Application
----------------------------------

The paster command line application will be installed as part of the
previous easy_install command. This application can be used to startup
the LWR server. This can be done by executing the following command::

    paster serve server.ini

Alternatively, the server may be ran as a daemon via the command::

    paster serve server.ini start

When running as daemon, the server may be stopped with the following command::

    paster serve server.ini stop

Remember if you setup a virtual environment for the LWR you will need
to activate this before executing these commands.

--------------------------------------
Configuring the LWR Server Application
--------------------------------------

Rename the ``server.ini.sample`` file distributed with LWR to ``server.ini``,
and edit the values therein to configure the server
application. Default values are specified for all configuration
options that will work if LWR is running on the same host as
Galaxy. However, the parameter "host" must be specified for remote
submissions to the LWR server to run properly. The ``server.ini`` file
contains documentation for many configuration parameters you may want
to modify.

Some advanced configuration topics are discussed below.

Securing the LWR
----------------

Out of the box the LWR essentially allows anyone with network access
to the LWR server to execute arbitrary code and read and write any
files the web server can. Hence, in most settings steps should be
taken to secure the LWR server.

The LWR can be configured to use SSL and to require the client
(i.e. Galaxy) to pass along a private token authorizing use.

To enable SSL, set the option ``ssl_pem`` in ``server.ini``. This
parameter should reference an OpenSSL certificate file for use by the
Python paste server. This parameter can be set to ``*`` to
automatically generate such a certificate. Such a certificate can
manually be generated by the following method::

    $ openssl genrsa 1024 > host.key
    $ chmod 400 host.key
    $ openssl req -new -x509 -nodes -sha1 -days 365  \
              -key host.key > host.cert
    $ cat host.cert host.key > host.pem
    $ chmod 400 host.pem

More information can be found in the `paste httpserver documentation
<http://pythonpaste.org/modules/httpserver.html>`_.

To specify a private token, simply set to ``private_key`` to some long
random string in ``server.ini``.

Once SSL has been enabled a private token configured, Galaxy job
runners need to be updated to use https and pass along this same
token.::

    <tool_id> = lwr://https://<lwr_private_token>@<lwr_host>:<lwr_port>


Multiple Job Managers (Queues)
------------------------------

An LWR server can be configured for multiple queues with different
properties (e.g. maximum number of concurrent jobs).

To configure queues, rename the file ``job_managers.ini.sample``
distributed with the LWR to ``job_managers.ini``. And comment the line
``#job_managers_config = job_managers.ini`` in ``server.ini``.

-------
Testing
-------

A simple sanity test can be run against a running LWR server by
executing the following command (replace the URL command with the URL
of your running LWR application)::

    python run_client_tests.py --url=http://localhost:8913

This project is distributed with unit tests, the following command
will install the needed python components to run these tests.::

    easy_install nose coverage webtest

The following command will then run these tests::

    nosetests

The following command will then produce a coverage report
corresponding to this test and place it in the coverage_html_report
subdirectory of this project.::

    coverage html
<commit_msg>Documentation improvements.<commit_after>===
LWR
===

This project is a Python server application that allows a 
`Galaxy <http://galaxyproject.org>`_ server to run jobs on remote 
systems (including Windows) without requiring a shared mounted 
file systems. Input files, scripts, and
config files are transferred to the remote system, the job is
executed, and the results are transferred back to the Galaxy server.

------------------
Configuring Galaxy
------------------

A Galaxy tool can be configured to be executed remotely via LWR by
adding a line to the ``universe.ini`` file under the ``galaxy:tool_runners``
section with the format::

    <tool_id> = lwr://http://<lwr_host>:<lwr_port>

As an example, if a host named remotehost is running the LWR server
application on port ``8913``, then the tool with id ``test_tool`` can be
configured to run remotely on remotehost by adding the following line
to ``universe.ini``::

    test_tool = lwr://http://remotehost:8913

Remember this must be added after the ``[galaxy:tool_runners]`` header in
the ``universe.ini`` file.

---------------
Downloading LWR
---------------

The LWR server application is distributed as a Python project and can
be obtained via mercurial from bitbucket.org using the following
command::

    hg clone http://bitbucket.org/jmchilton/lwr

----------------
LWR Dependencies
----------------

Several Python packages must be installed to run the LWR server. These
can either be installed into a Python ``virtualenv`` or into your
system wide Python environment using ``easy_install``. Instructions
for both are outlined below:

virtualenv
----------

The script ``setup_venv.sh`` distributed with the LWR server is a
short-cut for \*nix machines to setup a Python environment (including
the installation of virtualenv). Full details for installation
suitable for \*nix or Windows are as follows.

1. Install `virtualenv <http://www.virtualenv.org/en/latest/#installation>`_::

    pip install virtualenv

2. Create a new Python environment::

    virtualenv -q --no-site-packages .venv

3. Activate environment (varies by OS). 

From a Linux or MacOS terminal::

    . .venv/bin/activate

From a Windows terminal::

    .venv\Scripts\activate

4. Install required dependencies into this virtual environment::

    pip install -r tools/pip-requires

easy_install
------------

Install python setuptools for your platform, more details on how to do
this can be found `here <http://pypi.python.org/pypi/setuptools>`_.

The ``easy_install`` command line application will be installed as part of
setuptools. Use the following command to install the needed packages
via ``easy_install``::

    easy_install paste wsgiutils PasteScript PasteDeploy simplejson webob pyOpenSSL

``pyOpenSSL`` is only required if LWR is configured to use HTTPS/SSL.

----------------------------------
Running the LWR Server Application
----------------------------------

The paster command line application will be installed as part of the
previous easy_install command. This application can be used to startup
the LWR server. This can be done by executing the following command::

    paster serve server.ini

Alternatively, the server may be ran as a daemon via the command::

    paster serve server.ini start

When running as daemon, the server may be stopped with the following command::

    paster serve server.ini stop

Remember if you setup a virtual environment for the LWR you will need
to activate this before executing these commands.

--------------------------------------
Configuring the LWR Server Application
--------------------------------------

Rename the ``server.ini.sample`` file distributed with LWR to ``server.ini``,
and edit the values therein to configure the server
application. Default values are specified for all configuration
options that will work if LWR is running on the same host as
Galaxy. However, the parameter "host" must be specified for remote
submissions to the LWR server to run properly. The ``server.ini`` file
contains documentation for many configuration parameters you may want
to modify.

Some advanced configuration topics are discussed below.


Securing the LWR
----------------

Out of the box the LWR essentially allows anyone with network access
to the LWR server to execute arbitrary code and read and write any
files the web server can. Hence, in most settings steps should be
taken to secure the LWR server.

The LWR can be configured to use SSL and to require the client
(i.e. Galaxy) to pass along a private token authorizing use.

To enable SSL, set the option ``ssl_pem`` in ``server.ini``. This
parameter should reference an OpenSSL certificate file for use by the
Python paste server. This parameter can be set to ``*`` to
automatically generate such a certificate. Such a certificate can
manually be generated by the following method::

    $ openssl genrsa 1024 > host.key
    $ chmod 400 host.key
    $ openssl req -new -x509 -nodes -sha1 -days 365  \
              -key host.key > host.cert
    $ cat host.cert host.key > host.pem
    $ chmod 400 host.pem

More information can be found in the `paste httpserver documentation
<http://pythonpaste.org/modules/httpserver.html>`_.

To specify a private token, simply set to ``private_key`` to some long
random string in ``server.ini``.

Once SSL has been enabled a private token configured, Galaxy job
runners need to be updated to use https and pass along this same
token.::

    <tool_id> = lwr://https://<lwr_private_token>@<lwr_host>:<lwr_port>



Multiple Job Managers (Queues)
------------------------------

An LWR server can be configured for multiple queues with different
properties (e.g. maximum number of concurrent jobs).

To configure queues, rename the file ``job_managers.ini.sample``
distributed with the LWR to ``job_managers.ini``. And comment the line
``#job_managers_config = job_managers.ini`` in ``server.ini``.

-------
Testing
-------

A simple sanity test can be run against a running LWR server by
executing the following command (replace the URL command with the URL
of your running LWR application)::

    python run_client_tests.py --url=http://localhost:8913

This project is distributed with unit tests, the following command
will install the needed python components to run these tests.::

    easy_install nose coverage webtest

The following command will then run these tests::

    nosetests

The following command will then produce a coverage report
corresponding to this test and place it in the coverage_html_report
subdirectory of this project.::

    coverage html

{'fraction_non_alphanumeric': 0.10824585425224403, 'fraction_numerical': 0.005020538566864446, 'mean_word_length': 3.314735805710535, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 31, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7500339', 'n_tokens_mistral': 3469, 'n_tokens_neox': 3296, 'n_words': 1713}