<commit_before>.. highlight:: console

.. _cloudInstallation:

Cloud installation
==================

This section details how to properly set up Toil and its dependencies in various cloud environments.

.. _installationAWS:

Amazon Web Services
-------------------
Toil includes a native AWS provisioner that can be used to start :ref:`Autoscaling`
clusters. To provision static, non-autoscaling clusters we recommend using
CGCloud_.

.. _Toil_Provisioner:

Toil Provisioner
~~~~~~~~~~~~~~~~
The native Toil provisioner is included in Toil alongside the ``[aws]`` extra and
allows us to spin up a cluster without any external dependencies. It is built around the
Toil Appliance, a Docker image that bundles Toil and all its requirements,
e.g. Mesos. This makes deployment simple across platforms, and you can even
simulate a cluster locally (see :ref:`appliance_dev` for details).

When using the Toil provisioner, the appliance image will be automatically chosen
based on the pip installed version of Toil on your system. That choice can be
overriden by setting the environment variables ``TOIL_DOCKER_REGISTRY`` and ``TOIL_DOCKER_NAME`` or
``TOIL_APPLIANCE_SELF``. See :ref:`envars` for more information on these variables.

Using the provisioner to launch a Toil leader instance is simple::

    $ toil launch-cluster CLUSTER-NAME-HERE --nodeType=t2.micro \
       -z us-west-2a --keyPairName=your-AWS-key-pair-name

The cluster name is used to uniquely identify your cluster and will be used to
populate the instance's ``Name`` tag. In addition, the Toil provisioner will
automatically tag your cluster with an ``Owner`` tag that corresponds to your
keypair name to facilitate cost tracking.

The ``-z`` parameter is important since it specifies which EC2 availability
zone to launch the cluster in. Alternatively, you can specify this option
via the ``TOIL_AWS_ZONE`` environment variable. This is generally preferable
since it lets us avoid repeating the ``-z`` option for every subsequent
cluster command. We will assume this environment variable is set for the
rest of the tutorial. Note: the zone is different from an EC2 region. A
region corresponds to a geographical area like ``us-west-2 (Oregon)``, and<commit_msg>Add link to appliance development<commit_after>.. highlight:: console

.. _cloudInstallation:

Cloud installation
==================

This section details how to properly set up Toil and its dependencies in various cloud environments.

.. _installationAWS:

Amazon Web Services
-------------------
Toil includes a native AWS provisioner that can be used to start :ref:`Autoscaling`
clusters. To provision static, non-autoscaling clusters we recommend using
CGCloud_.

.. _Toil_Provisioner:

Toil Provisioner
~~~~~~~~~~~~~~~~
The native Toil provisioner is included in Toil alongside the ``[aws]`` extra and
allows us to spin up a cluster without any external dependencies. It is built around the
Toil Appliance, a Docker image that bundles Toil and all its requirements,
e.g. Mesos. This makes deployment simple across platforms, and you can even
simulate a cluster locally (see :ref:`appliance_dev` for details).

When using the Toil provisioner, the appliance image will be automatically chosen
based on the pip installed version of Toil on your system. That choice can be
overriden by setting the environment variables ``TOIL_DOCKER_REGISTRY`` and ``TOIL_DOCKER_NAME`` or
``TOIL_APPLIANCE_SELF``. See :ref:`envars` for more information on these variables. If
you are developing autoscaling and want to test and build your own
appliance have a look at :ref:`appliance_dev`.

Using the provisioner to launch a Toil leader instance is simple::

    $ toil launch-cluster CLUSTER-NAME-HERE --nodeType=t2.micro \
       -z us-west-2a --keyPairName=your-AWS-key-pair-name

The cluster name is used to uniquely identify your cluster and will be used to
populate the instance's ``Name`` tag. In addition, the Toil provisioner will
automatically tag your cluster with an ``Owner`` tag that corresponds to your
keypair name to facilitate cost tracking.

The ``-z`` parameter is important since it specifies which EC2 availability
zone to launch the cluster in. Alternatively, you can specify this option
via the ``TOIL_AWS_ZONE`` environment variable. This is generally preferable
since it lets us avoid repeating the ``-z`` option for every subsequent
cluster command. We will assume this environment variable is set for the
rest of the tutorial. Note: the zone is different from an EC2 region. A
region corresponds to a geographical area like ``us-west-2 (Oregon)``, and
{'fraction_non_alphanumeric': 0.07990420204659264, 'fraction_numerical': 0.002177226213803614, 'mean_word_length': 4.609279609279609, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '78655', 'n_tokens_mistral': 1214, 'n_tokens_neox': 1167, 'n_words': 624}