<commit_before>Housekeeping
------------
 1) Remove any old archives from:
        /var/tmp/rsr/data_archives
        /var/git/archives
 2) Remove any old backup databases through the mysql command-line tool:
        rsrdb_xxx_timestamp
 3) Remove unused system packages:
        sudo aptitude purge python-libxml2, pythonlibxslt1 and python-mysqldb
 4) List any broken packages then purge them if they're not needed:
        sudo aptitude install
 5) Update any out of date system packages with aptitude
        sudo aptitude update
        sudo aptitude safe-upgrade
        sudo aptitude install (to fix any missing dependencies, etc.)

Pre-installation tidy-up
------------------------
 1) Clear out the /var/virtualenvs dir: (the virtualenv will be rebuild with Python 2.7)
        - remove any pip_install*.log files
        - remove any virtualenvwrapper helper scripts (post*, pre*, hook.log, etc.)
        - remove any old virtualenvs *except* the last release env
 2) Run pip freeze to see which system Python packages are installed.
 3) Remove any extraneous packages, esp. the following:
        sudo pip uninstall supervisor meld3 mercurial
        sudo pip uninstall virtualenvwrapper virtual
    The only packges that should remain are: distribute, wsgiref and mod-python (though this will be uninstalled later)
 4) Ensure /usr/bin does not have any of the following binaries:
        pip (this will have to be removed)
        virtualenv
        virtualenvwrapper.sh

    Note: /usr/bin/pip-2.5 can still be used later if necessary else delete this as well

Python 2.7 installation
-----------------------
 1) Install Pythonbrew:
        cd /some/temp/dir
        export PYTHONBREW_ROOT=/usr/local/pythonbrew
        curl -kLO http://xrl.us/pythonbrewinstall
        chmod +x pythonbrewinstall
        sudo ./pythonbrewinstall
 2) Create symlink for pythonbrew:
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/bin/pythonbrew
 3) Disconnect and login again
 4) Install Python 2.7:
        sudo PYTHONBREW_ROOT=/usr/local/pythonbrew pythonbrew install --no-setuptools --verbose 2.7.3 --configure=--enable-shared
 5) Enable and link Python 2.7:
        sudo PYTHONBREW_ROOT=/usr/local/pythonbrew pythonbrew switch 2.7.3
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/python
        cd /lib
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/lib/libpython2.7.so.1.0
 6) Disconnect and login again
 7) Install distribute and pip:
        sudo bash
        python --version (this should be 2.7.3 -- if not, check that you've linked the installed version in /usr/local/bin)
        cd /some/temp/path
        curl -kLO http://python-distribute.org/distribute_setup.py
        sudo python distribute_setup.py
        wget -nv --no-check-certificate https://raw.github.com/pypa/pip/master/contrib/get-pip.py
        sudo python get-pip.py
10) Create a symlink in /usr/local/bin for pip
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/pip /usr/local/bin/pip
11) Use pip freeze to check the installation, which should list these packages:
        distribute==0.6.27
        wsgiref==0.1.2

Reinstall the virtualenv tools
------------------------------
 1) From your local machine, checkout the develop or release branch
 2) Run scripts/deployment/builders/rebuild_osx_dev_env.sh
 3) Update the system Python packges on the remote server:
        workon rsr_develop
        cd scripts/deployment/execution/steps
        ./2_update_system_python_packages.py "preconfigured:test" [or "preconfigured:uat", etc.]
 4) Back on the host, pip freeze should list:
        distribute==0.6.27
        virtualenv==1.7.2
        virtualenv-clone==0.2.4
        virtualenvwrapper==3.5
        wsgiref==0.1.2
 5) Create a symlink for virtualenv and virtualenvwrapper:
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/virtualenv
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/virtualenvwrapper.sh
 6) Reboot the deployment host:
        sudo reboot

Install mod_wsgi
----------------
 1) List the installed Apache packages with:
        aptitude search apache | grep ^i
    This should be:
        apache2
        apache2-mpm-prefork (since libapache2-mod-php5 depends on this package)
        apache2-utils
        apache2.2-common
        libapache2-mod-php5
    Install or replace any missing/incorrect modules
 2) Install Apache packages for mod_wsgi compilation:
        sudo aptitude install apache2-prefork-dev
 3) Uninstall the mod_python package if necessary: (since we're replacing this with mod_wsgi)
        sudo aptitude purge libapache2-mod-python
 4) Install mod_wsgi:
        cd /some/temp/dir
        wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz
        tar -xvf mod_wsgi-3.3.tar.gz
        cd mod_wsgi-3.3
        ./configure --with-python=/usr/local/bin/python --with-apxs=/usr/bin/apxs2
        make
        sudo make install
 5) Create a mod_wsgi loader file for Apache:
        cd /etc/apache2/mods-available
        sudo vi mod_wsgi.load
    with the following contents:

LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

 6) Update the Apache config files as follows: (adjust any paths as necessary)
    a) create copies of the /etc/apache2/sites-available files in /usr/local/etc/akvo/apache2
    b) edit the copied files in /usr/local/etc/akvo/apache2 only
    c) add the following directives to the top of the file, above any VirtualHost sections:

WSGIScriptAlias / /var/git/current/akvo/handler.py
WSGIPythonPath /var/git/current:/var/virtualenvs/current/lib/python2.7/site-packages

    d) withing the VirtualHost section and below the static content Directory directives, add the following:

<Directory /var/git/current/akvo>
    <Files handler.py>
        Order deny,allow
        Allow from all
    </Files>
</Directory>
 
    e) remove any obsolete Location sections for the old akvo_modpython handler
    f) ensure the access log is configured as follows:

CustomLog /var/log/apache2/access.log combined env=!forwarded
CustomLog /var/log/apache2/access.log proxy env=forwarded

    g) create symlinks for the new config files
        cd /etc/apache2/sites-available
        sudo rm default
        sudo rm akvotest.org.conf [or variant]
        sudo ln -s /usr/local/etc/akvo/apache2/default
        sudo ln -s /usr/local/etc/akvo/apache2/akvotest.org.conf

 7) Enable the mod_wsgi module and restart Apache:
        sudo a2enmod mod_wsgi
        sudo /etc/init.d/apache2 force-reload<commit_msg>[#55] Updated the Apache config details for the mod_wsgi module setup.<commit_after>Housekeeping
------------
 1) Remove any old archives from:
        /var/tmp/rsr/data_archives
        /var/git/archives
 2) Remove any old backup databases through the mysql command-line tool:
        rsrdb_xxx_timestamp
 3) Remove unused system packages:
        sudo aptitude purge python-libxml2, pythonlibxslt1 and python-mysqldb
 4) List any broken packages then purge them if they're not needed:
        sudo aptitude install
 5) Update any out of date system packages with aptitude
        sudo aptitude update
        sudo aptitude safe-upgrade
        sudo aptitude install (to fix any missing dependencies, etc.)

Pre-installation tidy-up
------------------------
 1) Clear out the /var/virtualenvs dir: (the virtualenv will be rebuild with Python 2.7)
        - remove any pip_install*.log files
        - remove any virtualenvwrapper helper scripts (post*, pre*, hook.log, etc.)
        - remove any old virtualenvs *except* the last release env
 2) Run pip freeze to see which system Python packages are installed.
 3) Remove any extraneous packages, esp. the following:
        sudo pip uninstall supervisor meld3 mercurial
        sudo pip uninstall virtualenvwrapper virtual
    The only packges that should remain are: distribute, wsgiref and mod-python (though this will be uninstalled later)
 4) Ensure /usr/bin does not have any of the following binaries:
        pip (this will have to be removed)
        virtualenv
        virtualenvwrapper.sh

    Note: /usr/bin/pip-2.5 can still be used later if necessary else delete this as well

Python 2.7 installation
-----------------------
 1) Install Pythonbrew:
        cd /some/temp/dir
        export PYTHONBREW_ROOT=/usr/local/pythonbrew
        curl -kLO http://xrl.us/pythonbrewinstall
        chmod +x pythonbrewinstall
        sudo ./pythonbrewinstall
 2) Create symlink for pythonbrew:
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/bin/pythonbrew
 3) Disconnect and login again
 4) Install Python 2.7:
        sudo PYTHONBREW_ROOT=/usr/local/pythonbrew pythonbrew install --no-setuptools --verbose 2.7.3 --configure=--enable-shared
 5) Enable and link Python 2.7:
        sudo PYTHONBREW_ROOT=/usr/local/pythonbrew pythonbrew switch 2.7.3
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/python
        cd /lib
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/lib/libpython2.7.so.1.0
 6) Disconnect and login again
 7) Install distribute and pip:
        sudo bash
        python --version (this should be 2.7.3 -- if not, check that you've linked the installed version in /usr/local/bin)
        cd /some/temp/path
        curl -kLO http://python-distribute.org/distribute_setup.py
        sudo python distribute_setup.py
        wget -nv --no-check-certificate https://raw.github.com/pypa/pip/master/contrib/get-pip.py
        sudo python get-pip.py
10) Create a symlink in /usr/local/bin for pip
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/pip /usr/local/bin/pip
11) Use pip freeze to check the installation, which should list these packages:
        distribute==0.6.27
        wsgiref==0.1.2

Reinstall the virtualenv tools
------------------------------
 1) From your local machine, checkout the develop or release branch
 2) Run scripts/deployment/builders/rebuild_osx_dev_env.sh
 3) Update the system Python packges on the remote server:
        workon rsr_develop
        cd scripts/deployment/execution/steps
        ./2_update_system_python_packages.py "preconfigured:test" [or "preconfigured:live", etc.]
 4) Back on the host, pip freeze should list:
        distribute==0.6.27
        virtualenv==1.7.2
        virtualenv-clone==0.2.4
        virtualenvwrapper==3.5
        wsgiref==0.1.2
 5) Create a symlink for virtualenv and virtualenvwrapper:
        cd /usr/local/bin
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/virtualenv
        sudo ln -s /usr/local/pythonbrew/pythons/Python-2.7.3/bin/virtualenvwrapper.sh
 6) Reboot the deployment host:
        sudo reboot

Install mod_wsgi
----------------
 1) List the installed Apache packages with:
        aptitude search apache | grep ^i
    This should be:
        apache2
        apache2-mpm-prefork (since libapache2-mod-php5 depends on this package)
        apache2-utils
        apache2.2-common
        libapache2-mod-php5
    Install or replace any missing/incorrect modules
 2) Install Apache packages for mod_wsgi compilation:
        sudo aptitude install apache2-prefork-dev
 3) Uninstall the mod_python package if necessary: (since we're replacing this with mod_wsgi)
        sudo aptitude purge libapache2-mod-python
 4) Install mod_wsgi:
        cd /some/temp/dir
        wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz
        tar -xvf mod_wsgi-3.3.tar.gz
        cd mod_wsgi-3.3
        ./configure --with-python=/usr/local/bin/python --with-apxs=/usr/bin/apxs2
        make
        sudo make install
 5) Create a mod_wsgi loader file for Apache:
        cd /etc/apache2/mods-available
        sudo vi mod_wsgi.load
    with the following contents:

LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

 6) Update the Apache config files as follows: (adjust any paths as necessary)
    a) create copies of the /etc/apache2/sites-available files in /usr/local/etc/akvo/apache2
       - rename the default file to default.conf
    b) edit the copied files in /usr/local/etc/akvo/apache2 only
    c) add the following directive to the top of the file, above any VirtualHost sections:

WSGIPythonPath /var/git/current:/var/virtualenvs/current/lib/python2.7/site-packages

    d) withing the VirtualHost section and below the static content Directory directives, add the following:

WSGIScriptAlias /api /var/git/current/akvo/handler.py
WSGIScriptAlias / /var/git/current/akvo/handler.py
<Directory /var/git/current/akvo>
    <Files handler.py>
        Order deny,allow
        Allow from all
    </Files>
</Directory>
 
    e) remove any obsolete Location sections for the old akvo_modpython handler
    f) ensure the access log is configured as follows:

CustomLog /var/log/apache2/access.log combined env=!forwarded
CustomLog /var/log/apache2/access.log proxy env=forwarded

    g) create symlinks for the new config files
        cd /etc/apache2/sites-available
        sudo rm default
        sudo rm akvotest.org.conf [or akvoapp.org.conf, etc]
        sudo ln -s /usr/local/etc/akvo/apache2/default.conf
        sudo ln -s /usr/local/etc/akvo/apache2/akvotest.org.conf [or akvoapp.org.conf, etc]
    h) ensure the symlinks in /etc/apache2/sites-enabled are unbroken and valid
 7) Enable the mod_wsgi module and restart Apache:
        sudo a2enmod mod_wsgi
        sudo /etc/init.d/apache2 force-reload
{'fraction_non_alphanumeric': 0.09379402387484244, 'fraction_numerical': 0.01935196856231927, 'mean_word_length': 2.608346709470305, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1881138', 'n_tokens_mistral': 4305, 'n_tokens_neox': 4005, 'n_words': 1340}