<commit_before>Installation
============

Because this tool uses the `FuXi`_ inference engine, it requires a
certain branch of `RDFLib`, namelay the `layercake
branch`_. Installing is nonetheless easy enough. It is recommended to
make use of `virtualenv`_ to create a python environment for this tool.

So a typical installation recipe might be to do:

.. code-block:: sh

    ## set up the environment
    pip install virtualenv
    virtualenv --no-site-packages --distribute curation
    . ./curation/bin/activate

    cd curation

    ## install the layercake branch of RDFLib and other prerequisites
    wget --no-check-certificate \
        https://bitbucket.org/okfn/curate/raw/tip/pip-requirements.txt
    pip install -r pip-requirements.txt

    ## this is needed for talking to SPARQL endpoints and cannot
    ## be installed with pip
    easy_install 4Suite-XML

    ## install the curate tool
    pip install curate
    
.. _FuXi: http://code.google.com/p/fuxi
.. _RDFLib: http://www.rdflib.net/
.. _layercake branch: http://code.google.com/p/python-dlp/wiki/LayerCakePythonDivergence
.. _virtualenv: http://pypi.python.org/pypi/virtualenv<commit_msg>put back installation of pyparsing explicitly in docs -- why doesn't this work from pip-requirements.txt\?<commit_after>Installation
============

Because this tool uses the `FuXi`_ inference engine, it requires a
certain branch of `RDFLib`, namelay the `layercake
branch`_. Installing is nonetheless easy enough. It is recommended to
make use of `virtualenv`_ to create a python environment for this tool.

So a typical installation recipe might be to do:

.. code-block:: sh

    ## set up the environment
    pip install virtualenv
    virtualenv --no-site-packages --distribute curation
    . ./curation/bin/activate

    cd curation

    ## install the layercake branch of RDFLib and other prerequisites
    pip install pyparsing
    wget --no-check-certificate \
        https://bitbucket.org/okfn/curate/raw/tip/pip-requirements.txt
    pip install -r pip-requirements.txt

    ## this is needed for talking to SPARQL endpoints and cannot
    ## be installed with pip
    easy_install 4Suite-XML

    ## install the curate tool
    pip install curate
    
.. _FuXi: http://code.google.com/p/fuxi
.. _RDFLib: http://www.rdflib.net/
.. _layercake branch: http://code.google.com/p/python-dlp/wiki/LayerCakePythonDivergence
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
{'fraction_non_alphanumeric': 0.09864920180106426, 'fraction_numerical': 0.0008186655751125665, 'mean_word_length': 3.325663716814159, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1266397', 'n_tokens_mistral': 764, 'n_tokens_neox': 721, 'n_words': 245}