<commit_before>### Dependencies

 * Make sure that Python 2.7 is installed, otherwise install from http://python.org
 * Install Twisted Matrix server from https://twistedmatrix.com/trac/
 * Install element tree "lxml" from http://lxml.de/index.html
 * Install "sass" from http://sass-lang.com
 * Install "boto" from https://aws.amazon.com/sdkforpython/

### Optional Dependencies

 * Install "cjson" from http://sourceforge.net/projects/cjson/
 * Make sure "cStringIO" is installed, see http://pydoc.org/2.4.1/cStringIO.html
 * Make sure "cPickle" is installed, see http://pymotw.com/2/pickle 

### Installation

 * Add xierpa3.pth to Python site-packages, referring to this git Xierpa3 install
 * Copy xierpa3/constants/config/COPYTO-config.py to xierpa3/constants/config.config.py and adjust the parameters and access keys to your own settings. Make sure this config.py never writes back into git (future development is to store this file in your local user folder).

### Databases

A variety of database connections is possible: local and online SQL databases, Amazon database services and S3. The access codes and IP/URL's need to be filled into the config.py or defined locally in the site application.

-----------------------------------------------------------------------------<commit_msg>Update INSTALL.md<commit_after>## Installing Xierpa

Various possibilities are available to install Python packages Xierpa depends on, as well as third-party applications. On OSX, you can either get the Python dependencies through easy_install, pip, or directly from the Python package index. Make sure that packages that have C/C++ bindings are compiled with the same version of GCC as the version of Python you're using. It is also possible to pull most of the Python dependencies from MacPorts or Homebrew. On Linux, most packages are available from package managers such as apt-get / aptitude or yum.

### Installation

 * Add xierpa3.pth to Python site-packages, referring to the folder where Xierpa3 has been cloned from git. For example, the site-packages for a prepackaged Python on a recent OSX machine can be found here: /Library/Python/2.7/site-packages.
 * Copy xierpa3/constants/config/COPYTO-config.py to xierpa3/constants/config.config.py and adjust the parameters and access keys to your own settings. Make sure this config.py never writes back into git (future development is to store this file in your local user folder).

### Dependencies

 * Make sure that Python 2.7 is installed, otherwise get it from http://python.org. Recent versions of OSX and Linux come with a completely installed Python environment.
 * Install the Twisted Matrix server, https://twistedmatrix.com/trac/ or https://pypi.python.org/pypi/Twisted
 * Install element tree "lxml", http://lxml.de/index.html or https://pypi.python.org/pypi/lxml/, which needs the libxml2 and libxslt binaries.
 * Install "sass" from http://sass-lang.com. For this you first need to install Ruby and then get the sass gem like this:
   * sudo gem install sass
 * Install "boto" from https://aws.amazon.com/sdkforpython/ or https://pypi.python.org/pypi/boto

### Optional Dependencies

 * Install "cjson" from http://sourceforge.net/projects/cjson/ or https://pypi.python.org/pypi/python-cjson for (much) faster JSON parsing.
   * Make sure "cStringIO" is installed, see http://pydoc.org/2.4.1/cStringIO.html
   * Make sure "cPickle" is installed, see http://pymotw.com/2/pickle

### Databases

A variety of database connections is possible: local and online SQL databases, Amazon database services and S3. The access codes and IP/URL's need to be filled into the config.py or defined locally in the site application.

-----------------------------------------------------------------------------
{'fraction_non_alphanumeric': 0.121413390010627, 'fraction_numerical': 0.0066418703506907545, 'mean_word_length': 3.455621301775148, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2293551', 'n_tokens_mistral': 1033, 'n_tokens_neox': 994, 'n_words': 443}