<commit_before>*********************************************************
Multiple Data Sources, Automated Source Updating Tutorial
*********************************************************

In this tutorial, we will build the whole process, or "hub", which produces the data
for Taxonomy BioThings API, accessible at `<t.biothings.io>`. This API serves information
about species, lineage, etc... This "hub" is used to download, maintain up-to-date,
process, merge data. At the end of this process, an Elasticsearch index is created
containing all the data of interest, ready to be served as an API, using
Biothings SDK Web component (covered in another tutorial).
Taxonomy Biothings API code is avaiable at `<https://github.com/biothings/biothings.species>`

Prerequesites
^^^^^^^^^^^^^

BioThings SDK uses MongoDB as the "staging" storage backend for JSON objects before they are sent to 
Elasticsearch for indexing. You must a have working MongoDB instance you can connect to.
We'll also perform some basic commands. You must also have BioThings SDK installed (
``git clone https://github.com/biothings/biothings.api.git`` is usually enough, followed by
``pip install -r requirements.txt``). You may want to use ``virtualenv`` to isolate your installation.
Finally, BioThings SDK is written in python, so you must know some basics.

Configuration file
^^^^^^^^^^^^^^^^^^

Before starting to implement our hub, we first need to define a configuration file. A 
`config_common.py <https://github.com/biothings/biothings.species/blob/master/src/config_common.py>` file
contains all the required configuration variables, some **have** to be defined in your own application, other
**can** be overriden as needed.

Typically we will have to define the following:

* MongoDB connections parameters, ``DATA_SRC_*`` and ``DATA_TARGET_*`` parameters. They define connections to two different databases,<commit_msg>update install guideline<commit_after>*********************************************************
Multiple Data Sources, Automated Source Updating Tutorial
*********************************************************

In this tutorial, we will build the whole process, or "hub", which produces the data
for Taxonomy BioThings API, accessible at `<t.biothings.io>`. This API serves information
about species, lineage, etc... This "hub" is used to download, maintain up-to-date,
process, merge data. At the end of this process, an Elasticsearch index is created
containing all the data of interest, ready to be served as an API, using
Biothings SDK Web component (covered in another tutorial).
Taxonomy Biothings API code is avaiable at `<https://github.com/biothings/biothings.species>`

Prerequesites
^^^^^^^^^^^^^

BioThings SDK uses MongoDB as the "staging" storage backend for JSON objects before they are sent to 
Elasticsearch for indexing. You must a have working MongoDB instance you can connect to.
We'll also perform some basic commands.

You also have to install the latest stable BioThings SDK release, with pip from `PyPI <https://pypi.python.org/pypi/biothings>`_:

::

    pip install biothings

You can install the latest development version of BioThings SDK directly from our github repository like:

::

    pip install git+https://github.com/biothings/biothings.api.git#egg=biothings

Alternatively, you can download the source code, or clone the `BioThings SDK repository <http://github.com/biothings/biothings.api>`_ and run:

::

    python setup.py install

You may want to use ``virtualenv`` to isolate your installation.

Finally, BioThings SDK is written in python, so you must know some basics.

Configuration file
^^^^^^^^^^^^^^^^^^

Before starting to implement our hub, we first need to define a configuration file. A 
`config_common.py <https://github.com/biothings/biothings.species/blob/master/src/config_common.py>` file
contains all the required configuration variables, some **have** to be defined in your own application, other
**can** be overriden as needed.

Typically we will have to define the following:

* MongoDB connections parameters, ``DATA_SRC_*`` and ``DATA_TARGET_*`` parameters. They define connections to two different databases,
{'fraction_non_alphanumeric': 0.13636363636363635, 'fraction_numerical': 0.0, 'mean_word_length': 5.334848484848485, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6631172', 'n_tokens_mistral': 1112, 'n_tokens_neox': 1038, 'n_words': 505}