<commit_before># otindex

A pyramid + postgres implementation of a treestore index for Open Tree of Life.
Provides API access to the JSON files in the OpenTree 
[phylesystem](https://github.com/opentreeoflife/phylesystem) data store. 

These instructions assume you are doing development setup on your local
machine. For deploying otindex on a server, we have [ansible
playbook](https://github.com/OpenTreeOfLife/otindex_ansible).

## Tech used in development

* python 2.7.10
* [pyramid](http://www.pylonsproject.org/) v 1.5.7
* [postgres](http://www.postgresql.org/) v 9.5.2
* [peyotl](https://github.com/OpenTreeOfLife/peyotl): opentree python library for interacting with the opentree tree store

## Setup your postgres database
See the
[README](https://github.com/OpenTreeOfLife/otindex/blob/master/otindex/scripts/README.md)
file in that directory for more detailed setup information.

# Postgres setup ubuntu version (draft)
    sudo apt-get install postgresql  
    service postgresql start  
    sudo -u postgres createdb -O postgres -E utf8 otindex  
    sudo -u postgres createuser opentree -D -S -P  <commit_msg>Update README to describe requirements tracking.<commit_after># otindex

A pyramid + postgres implementation of a treestore index for Open Tree of Life.
Provides API access to the JSON files in the OpenTree 
[phylesystem](https://github.com/opentreeoflife/phylesystem) data store. 

These instructions assume you are doing development setup on your local
machine. For deploying otindex on a server, we have [ansible
playbook](https://github.com/OpenTreeOfLife/otindex_ansible).

## Tech used in development

* python 2.7.10
* [pyramid](http://www.pylonsproject.org/) v 1.5.7
* [postgres](http://www.postgresql.org/) v 9.5.2
* [peyotl](https://github.com/OpenTreeOfLife/peyotl): opentree python library for interacting with the opentree tree store

### Tracking dependencies and their versions
The top-level dependencies for otindex are listed in `requirements.in`. These
are generally listed without versions, so a simple installation or update will
fetch the latest stable version of everything.

In practice, peyotl will also be run from the same virtual environment, so its
dependencies will mix with those of otindex. Hopefully this won't cause
conflicts going forward.

For actual ("known good") versions and a full list of lower-level dependencies,
look in `requirements.txt`. Assuming these versions have undergone some
testing, using this file is the safest path for deployment.

This version of `requirements.txt` clearly separates top-level from
lower-level dependencies, and ignores packages introduced by peyotl and
dev tools. This is better for fool-proof deployment, assuming you follow
a similar practice for peyotl in this venv. To update this file, use
```sh
$VENV/bin/pip freeze –r requirements.in > requirements.txt
```

For more insight into the full dependency tree, use
[`pipdeptree`](https://pypi.org/project/pipdeptree/) to see all
installed packages and dependencies. You can save updates to `pipdeptree.out`.

## Setup your postgres database
See the
[README](https://github.com/OpenTreeOfLife/otindex/blob/master/otindex/scripts/README.md)
file in that directory for more detailed setup information.

# Postgres setup ubuntu version (draft)
    sudo apt-get install postgresql  
    service postgresql start  
    sudo -u postgres createdb -O postgres -E utf8 otindex  
    sudo -u postgres createuser opentree -D -S -P  
{'fraction_non_alphanumeric': 0.0798040910400461, 'fraction_numerical': 0.006338231057332181, 'mean_word_length': 4.502377179080824, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2902732', 'n_tokens_mistral': 1030, 'n_tokens_neox': 984, 'n_words': 404}