<commit_before>***********************************************
Single Data Source, No Source Updating Tutorial
***********************************************

The following tutorial shows a minimal use case for the BioThings SDK: creating a
high-performance, high-concurrency API from a single flat-file.  The BioThings SDK
is broadly divided into two components, the hub and the web.  The hub component is a
collection of tools to automate the downloading of source data files, the merging
of different sources, and the updating of the Elasticsearch index.  The web component
is a Tornado-based API app that subsequently serves data from this Elasticsearch index.

Prerequisites
^^^^^^^^^^^^^

Before starting, there are a few requirements that need to be installed and configured.

Python
======

The BioThings SDK requires `Python version 3.4 or higher <https://www.python.org/>`_ for full functionality.
We recommend installing all python dependencies into a `virtualenv <https://virtualenv.pypa.io/en/stable/>`_.

BioThings SDK
=============

Either install from source, like:

.. code-block:: bash
    
    git clone https://github.com/biothings/biothings.api.git
    cd biothings.api
    python setup.py install

or use pip, like:

.. code-block:: bash

    pip install git+https://github.com/biothings/biothings.api.git#egg=biothings

Elasticsearch
=============

BioThings APIs currently serve data from an Elasticsearch index, so Elasticsearch is a requirement.
Install Elasticsearch 2.4 either `directly <https://www.elastic.co/guide/en/elasticsearch/reference/2.4/_installation.html>`_,
or as a `docker container <https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html>`_.

Configure Elasticsearch
+++++++++++++++++++++++

To configure Elasticsearch, execute the following commands as su.

.. code-block:: bash

    echo 'http.enabled: True' >> /etc/elasticsearch/elasticsearch.yml
    echo 'network.host: "0.0.0.0"' >> /etc/elasticsearch/elasticsearch.yml


PharmGKB Gene
^^^^^^^^^^^^^

Once all prerequisites have been installed, the data loading step can begin.
Consider the following script, which defines a "load_data" function that parses
the `PharmGKB gene flat file <https://api.pharmgkb.org/v1/download/file/data/genes.zip>`_ and then iterates through it, storing the results in an Elasticsearch index using biothings.utils.es.ESIndexer.

.. code-block:: python

    In [1]: import re

    In [2]: from biothings.utils.es import ESIndexer

    In [3]: def load_data(f):
       ...:     for (i, line) in enumerate(f):
       ...:         line = line.strip('\n')<commit_msg>Update single_source_tutorial.rst<commit_after>***********************************************
Single Data Source, No Source Updating Tutorial
***********************************************

The following tutorial shows a minimal use case for the BioThings SDK: creating a
high-performance, high-concurrency API from a single flat-file.  The BioThings SDK
is broadly divided into two components, the hub and the web.  The hub component is a
collection of tools to automate the downloading of source data files, the merging
of different sources, and the updating of the Elasticsearch index.  The web component
is a Tornado-based API app that subsequently serves data from this Elasticsearch index.

Prerequisites
^^^^^^^^^^^^^

Before starting, there are a few requirements that need to be installed and configured.

Python
======

The BioThings SDK requires `Python version 3.4 or higher <https://www.python.org/>`_ for full functionality.
We recommend installing all python dependencies into a `virtualenv <https://virtualenv.pypa.io/en/stable/>`_.

BioThings SDK
=============

Either install from source, like:

.. code-block:: bash
    
    git clone https://github.com/biothings/biothings.api.git
    cd biothings.api
    python setup.py install

or use pip, like:

.. code-block:: bash

    pip install git+https://github.com/biothings/biothings.api.git#egg=biothings

Elasticsearch
=============

BioThings APIs currently serve data from an Elasticsearch index, so Elasticsearch is a requirement.
Install Elasticsearch 2.4 either `directly <https://www.elastic.co/guide/en/elasticsearch/reference/2.4/_installation.html>`_,
or as a `docker container <https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html>`_.

Configure Elasticsearch
+++++++++++++++++++++++

To configure Elasticsearch, execute the following commands as su (this guide was created on Ubuntu 16.04, the exact location of elasticsearch.yml may vary in other platforms).

.. code-block:: bash

    echo 'http.enabled: True' >> /etc/elasticsearch/elasticsearch.yml
    echo 'network.host: "0.0.0.0"' >> /etc/elasticsearch/elasticsearch.yml


PharmGKB Gene
^^^^^^^^^^^^^

Once all prerequisites have been installed, the data loading step can begin.
Consider the following script, which defines a "load_data" function that parses
the `PharmGKB gene flat file <https://api.pharmgkb.org/v1/download/file/data/genes.zip>`_ and then iterates through it, storing the results in an Elasticsearch index using biothings.utils.es.ESIndexer.

.. code-block:: python

    In [1]: import re

    In [2]: from biothings.utils.es import ESIndexer

    In [3]: def load_data(f):
       ...:     for (i, line) in enumerate(f):
       ...:         line = line.strip('\n')
{'fraction_non_alphanumeric': 0.14626865671641792, 'fraction_numerical': 0.005970149253731343, 'mean_word_length': 4.307920792079208, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 21, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1193810', 'n_tokens_mistral': 1566, 'n_tokens_neox': 1470, 'n_words': 553}