<commit_before>## setup

I recommend cloning this repository inside an existing directory that's just
for this project.  The sqlite database will be created one directory up from
the repo, and that's also where I recommend you put your virtualenv.

So, create a virtualenv outside the repo, install packages from
`requirements.txt` for Python 2, or `p3req.txt` for Python 3, into it.  I
haven't picked a Python version (yet); if I haven't messed up, 2.7 and 3.4
should both work.  Trying to stay compatible with both.  See 'compatibility'
section below.

## compilation

So far the only code that needs compiling is the coffeescript in `app/coffee`
(and at time of writing, the resulting Javascript is still included in the<commit_msg>mention db_remake.py in README<commit_after>## setup

I recommend cloning this repository inside an existing directory that's just
for this project.  The sqlite database will be created one directory up from
the repo, and that's also where I recommend you put your virtualenv.

So, create a virtualenv outside the repo, install packages from
`requirements.txt` for Python 2, or `p3req.txt` for Python 3, into it.  I
haven't picked a Python version (yet); if I haven't messed up, 2.7 and 3.4
should both work.  Trying to stay compatible with both.  See 'compatibility'
section below.

Run `db_remake.py` to create the database (by default it will appear one
directory level higher than your repository root).

## compilation

So far the only code that needs compiling is the coffeescript in `app/coffee`
(and at time of writing, the resulting Javascript is still included in the
{'fraction_non_alphanumeric': 0.05521091811414392, 'fraction_numerical': 0.008684863523573201, 'mean_word_length': 4.505119453924915, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1476236', 'n_tokens_mistral': 456, 'n_tokens_neox': 423, 'n_words': 249}