<commit_before>======================
Installation and Setup
======================

Download and Install (All Version)
----------------------------------

This assumes you already have CKAN installed on this server in the same location as the docs mention or via a package install.  If this is correct you should be able to run the following as is::

    #go to the ckan source directory
    cd /usr/lib/ckan/default/src

    #clone the source
    sudo git clone https://github.com/okfn/datapusher.git

    #install the datapussher
    cd datapusher
    sudo /usr/lib/ckan/default/bin/python setup.py develop

    #copy the standard apache config file
    sudo cp deployment/datapusher /etc/apache2/sites-available/

    #copy the standard datapusher wsgi file
    sudo cp deployment/datapusher.wsgi /etc/ckan/

    #copy the standard datapusher settings.
    sudo cp deployment/datapusher_settings /etc/ckan/

    #open up port 8800 on apache where the datapusher accepts connections.
    sudo sh -c 'echo "NameVirtualHost *:8800" >> /etc/apache2/ports.conf'
    sudo sh -c 'echo "Listen 8800" >> /etc/apache2/ports.conf'

    #enable datapusher apache
    sudo a2ensite datapusher
<commit_msg>add configuration option to documentation<commit_after>======================
Installation and Setup
======================

Download and Install (All Version)
----------------------------------

This assumes you already have CKAN installed on this server in the same location as the docs mention or via a package install.  If this is correct you should be able to run the following as is::

    #go to the ckan source directory
    cd /usr/lib/ckan/default/src

    #clone the source
    sudo git clone https://github.com/okfn/datapusher.git

    #install the datapussher
    cd datapusher
    sudo /usr/lib/ckan/default/bin/python setup.py develop

    #copy the standard apache config file
    sudo cp deployment/datapusher /etc/apache2/sites-available/

    #copy the standard datapusher wsgi file
    sudo cp deployment/datapusher.wsgi /etc/ckan/

    #copy the standard datapusher settings.
    sudo cp deployment/datapusher_settings /etc/ckan/

    #open up port 8800 on apache where the datapusher accepts connections.
    sudo sh -c 'echo "NameVirtualHost *:8800" >> /etc/apache2/ports.conf'
    sudo sh -c 'echo "Listen 8800" >> /etc/apache2/ports.conf'

    #enable datapusher apache
    sudo a2ensite datapusher

These instructions setup the datapusher webservice.  In order to tell CKAN where this webservice is the following must be added to the [app:main] section of your CKAN configuration file located at ``/etc/ckan/default/production.ini``::

    ckan.datapusher.url = http://0.0.0.0:8800/

{'fraction_non_alphanumeric': 0.12347504621072089, 'fraction_numerical': 0.014787430683918669, 'mean_word_length': 3.1312977099236643, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3775914', 'n_tokens_mistral': 823, 'n_tokens_neox': 767, 'n_words': 298}