<commit_before>To install on Ubuntu for development:

You need python 2.7+, but *not* python 3+

1. install pip globally:
http://www.pip-installer.org/en/latest/installing.html#prerequisites

May need to install easy_install first: 
sudo apt-get install python-setuptools

Then install pip: 
sudo easy_install pip

2. install virtualenv:
http://www.virtualenv.org/en/latest/index.html

sudo pip install virtualenv 

3.  (highly recommended but not necessary):
install virtualenvwrapper:
http://virtualenvwrapper.readthedocs.org/en/latest/

sudo pip install virtualenvwrapper 
export WORKON_HOME=~/Envs
mkdir -p $WORKON_HOME
source /usr/local/bin/virtualenvwrapper.sh

Step 6 assumes virtualenvwrapper

4. unless you don't have it already: install git
sudo apt-get install git

5. We have had problems getting Pillow, one of the requirements, 
to handle jpegs without installing some external libraries.

It's gone from requirements.txt until we figure this out more.
But you will need either Pillow or PIL, the Python Imaging Library.

See 
http://pillow.readthedocs.org/en/latest/installation.html#simple-installation
for how to install Pillow on your operating system.

Note: <NAME> reports that on Debian, Pillow requires
libjpeg-dev and python2.7-dev.

Here's PIL: https://pypi.python.org/pypi/PIL/

6. create a virtualenv (assuming you installed virtualenvwrapper):
mkvirtualenv vn --system-site-packages 
workon vn
cdvirtualenv

7. clone the valuenetwork repo inside the vn virtualenv.
git clone https://github.com/valnet/valuenetwork.git
 
8. cd valuenetwork

9. pip install -r requirements.txt
<commit_msg>Update local install docs with experience on pinax requirements.<commit_after>To install on Ubuntu for development:

You need python 2.7+, but *not* python 3+

1. install pip globally:
http://www.pip-installer.org/en/latest/installing.html#prerequisites

May need to install easy_install first: 
sudo apt-get install python-setuptools

Then install pip: 
sudo easy_install pip

2. install virtualenv:
http://www.virtualenv.org/en/latest/index.html

sudo pip install virtualenv 

3.  (highly recommended but not necessary):
install virtualenvwrapper:
http://virtualenvwrapper.readthedocs.org/en/latest/

sudo pip install virtualenvwrapper 
export WORKON_HOME=~/Envs
mkdir -p $WORKON_HOME
source /usr/local/bin/virtualenvwrapper.sh

Step 6 assumes virtualenvwrapper

4. unless you don't have it already: install git
sudo apt-get install git

5. We have had problems getting Pillow, one of the requirements, 
to handle jpegs without installing some external libraries.

It's gone from requirements.txt until we figure this out more.
But you will need either Pillow or PIL, the Python Imaging Library.

See 
http://pillow.readthedocs.org/en/latest/installation.html#simple-installation
for how to install Pillow on your operating system.

Note: <NAME> reports that on Debian, Pillow requires
libjpeg-dev and python2.7-dev.

Here's PIL: https://pypi.python.org/pypi/PIL/

6. create a virtualenv (assuming you installed virtualenvwrapper):
mkvirtualenv vn --system-site-packages 
workon vn
cdvirtualenv

7. clone the valuenetwork repo inside the vn virtualenv.
git clone https://github.com/valnet/valuenetwork.git
 
8. cd valuenetwork

9. pip install -r requirements.txt

Note: If you get errors saying the repository is not trusted, try adding: --trusted-host dist.pinaxproject.com to the above command.

{'fraction_non_alphanumeric': 0.07617931438617052, 'fraction_numerical': 0.008789920890711983, 'mean_word_length': 4.542207792207792, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6047904', 'n_tokens_mistral': 1098, 'n_tokens_neox': 1024, 'n_words': 387}