<commit_before>= Clearinghouse Installation How-To =

This document explains how to do a fresh install of the Seattle Clearinghouse portal on
a machine that is running a Debian-like Linux operating system such as Ubuntu.

 * For info on deploying the production Seattle Clearinghouse code and for starting/restarting Apache, see [wiki:SeattleGeniProductionHttp]

----
[[TOC(inline)]]
----
== Install Dependencies ==
----

Clearinghouse requires at least the following software to be installed:
 
 * [http://www.python.org/ Python] -- the language Seattle Clearinghouse is written in
 * [http://mysql-python.sourceforge.net/ Python MySQLDB] -- the python mysql interface
 * [http://www.mysql.com/ MySQL] -- the database
 * [http://www.apache.org/ Apache] -- the web server
 * [http://www.modpython.org/ mod_python] -- necessary for interfacing with Django code
 * [http://www.openssl.org/ OpenSSL] -- necessary for `https://` support
 * [http://www.djangoproject.com/ Django] -- necessary to run Django code

Optionally,
 * [https://github.com/omab/django-social-auth Django Social Auth] -- for OpenID and Oauth support

Most of these can be installed through a package manager. For example, on a Debian-based system:

{{{
#!sh
$ sudo apt-get install apache2 libapache2-mod-python
$ sudo apt-get install mysql-server mysql-client python-mysqldb
$ sudo apt-get install ntp
$ sudo apt-get install openssl
}}}

The version of Django required is 1.3.5, which is available at https://www.djangoproject.com/download/ or through a package manager.
{{{
#!sh
$ pip install django==1.3.5
}}}
At this time we do not support Django version 1.4 or higher.



== Optional: Setup OpenID and OAuth ==
If you would like your Clearinghouse to support login not only through user accounts it manages itself, but ID/authentication services like OpenID and OAuth, or web services like Google, Facebook, or !GitHub, take a look at the [wiki:ClearinghouseSocialAuth social auth support instructions page].



== Create MySQL databases ==

You need two mysql databases and seperate users with access to each.
  * First database name: `seattlegeni`
  * Second database name: `keydb`

For simplicity you can make the database usernames correspond to the database names.

Here's an example of creating a database and a user:

{{{
#!sh
$ mysql -u root -p

mysql> create database seattlegeni;

mysql> GRANT ALL PRIVILEGES 
ON seattlegeni.* 
TO 'seattlegeni'@'localhost'
IDENTIFIED BY 'password';
}}}
where you would replace the 'password' with the password you want to use for the user 'seattlegeni'

'''Repeat the process''', substituting `keydb` for `seattlegeni`. Afterwards, type `\q` to leave the MySQL prompt:

{{{
#!sh
mysql> \q
Bye!
$
}}}



== Deploying and running Clearinghouse ==
In this section, we will deploy and run a copy of the Clearinghouse from your current user account in a temporary directory. This is mainly useful for testing. For an actual deployment, we recommend setting up a separate user account, e.g. named {{{clearinghouse}}}, and following the steps below as this user.

  * '''Initial preparation'''
    * Clone the Clearinghouse repository into your home directory, and let the initialize script fetch dependencies:
{{{
#!sh
git clone https://github.com/aaaaalbert/clearinghouse.git
cd clearinghouse/scripts
python initialize.py
}}}
    * Deploy all necessary files to a directory of your choice. (We'll use `/tmp/TARGET` in these instructions.) The directory you deploy to should be the name of a directory that does not yet exist. For example:
{{{
#!sh 
python ./clearinghouse/deploymentscripts/deploy_seattlegeni.py . /tmp/TARGET
}}} 
    (Note: For a live deployment, the path will likely be {{{/home/geni/live}}}).

    * Deploy the Repy runtime, following the instructions of the script called previously.
{{{
#!sh
mkdir /tmp/TARGET/clearinghouse/seattle
cd clearinghouse/scripts/dist
python build.py /tmp/TARGET/clearinghouse/seattle   # Use absolute target path!
}}}


    * Change to the `clearinghouse` directory that is in the directory you deployed
      to. For example:
{{{
#!sh 
cd /tmp/TARGET/clearinghouse
}}}
      '''Note: All future steps/instructions will assume you are in this directory.'''
  * '''Future tip'''
    * Starting all of the processes described below is simplified by using the script
      `deploymentscripts/start_seattlegeni_components.sh`, whose use is described in
      the section on [#Runningstart_seattlegeni_components.sh running start_seattlegeni_components.sh].
      For reference, we'll explain here how to start the individual components as well
      as include first-time configuration information. Note that when starting these
      manually, you need to start them in the order shown. Most importantly, first the
      lockserver, then the backend, then everything else. However, we're not going to
      describe the initial setup in this order.
  * '''Generating state keys'''
    Seven different state keys need to be created in order for all the backend scripts
    to work. The name of the seven key files should be:
    1. acceptdonation.publickey
    1. movingto_canonical.publickey
    1. canonical.publickey
    1. movingto_onepercentmanyevents.publickey
    1. onepercentmanyevents.publickey
    1. movingto_twopercent.publickey
    1. twopercent.publickey
    * Note that the privatekey of the key pairs is not necessary. To generate the keys 
      follow these instructions: 
      * Create a temporary folder.
      * Navigate to the trunk/ folder.
      * Run the command:
{{{
python preparetest.py TEMP_FOLDER
}}}
        where the temp folder is the folder created in step 1.
      * Navigate to the TEMP_FOLDER.
      * Run the command:
{{{
python generatekeys.py NAME_OF_YOUR_PUBKEY_PRIV_KEY
}}}
       An example would be:
{{{
python generatekeys.py acceptdonation
}}}
      * Generate a key pair for all four state keys.
      * Create a folder named statekeys in your /tmp/deploy/seattlegeni/node_state_transitions/
        folder and copy the seven publickeys over to that folder.
      * You can delete the privatekeys if you want to as we won't need them.
      * Delete the TEMP_FOLDER and all its content at this point if you want to.
  * '''Setup the website and start a development version of it'''
    * Be sure you've already created a mysql database for seattlegeni (e.g.
      called `seattlegeni`).
    * Edit website/settings.py
      * Specify the name of the seattlegeni
        database, as well as the database username, password, etc.
      * Set a long, random string the value of SECRET_KEY.
      * If this is a production launch, also set DEBUG to False,
        and uncomment and change the fields for sending ADMINS email.
    * Set your environment variables:
{{{
#!sh
export PYTHONPATH=$PYTHONPATH:/tmp/deploy:/tmp/deploy/seattle
export DJANGO_SETTINGS_MODULE='seattlegeni.website.settings'<commit_msg>Revision 8 of Archive/ClearinghouseInstallationWithDjango13<commit_after>= Clearinghouse Installation How-To =

This document explains how to do a fresh install of the Seattle Clearinghouse portal on
a machine that is running a Debian-like Linux operating system such as Ubuntu.

 * For info on deploying the production Seattle Clearinghouse code and for starting/restarting Apache, see [wiki:SeattleGeniProductionHttp]

----
[[TOC(inline)]]
----
== Install Dependencies ==
----

Clearinghouse requires at least the following software to be installed:
 
 * [http://www.python.org/ Python] -- the language Seattle Clearinghouse is written in
 * [http://mysql-python.sourceforge.net/ Python MySQLDB] -- the python mysql interface
 * [http://www.mysql.com/ MySQL] -- the database
 * [http://www.apache.org/ Apache] -- the web server
 * [http://www.modpython.org/ mod_python] -- necessary for interfacing with Django code
 * [http://www.openssl.org/ OpenSSL] -- necessary for `https://` support
 * [http://www.djangoproject.com/ Django] -- necessary to run Django code

Optionally,
 * [https://github.com/omab/django-social-auth Django Social Auth] -- for OpenID and Oauth support

Most of these can be installed through a package manager. For example, on a Debian-based system:

{{{
#!sh
$ sudo apt-get install apache2 libapache2-mod-python
$ sudo apt-get install mysql-server mysql-client python-mysqldb
$ sudo apt-get install ntp
$ sudo apt-get install openssl
}}}

The version of Django required is 1.3.5, which is available at https://www.djangoproject.com/download/ or through a package manager.
{{{
#!sh
$ pip install django==1.3.5
}}}
At this time we do not support Django version 1.4 or higher.



== Optional: Setup OpenID and OAuth ==
If you would like your Clearinghouse to support login not only through user accounts it manages itself, but ID/authentication services like OpenID and OAuth, or web services like Google, Facebook, or !GitHub, take a look at the [wiki:ClearinghouseSocialAuth social auth support instructions page].



== Create MySQL databases ==

You need two mysql databases and seperate users with access to each.
  * First database name: `seattlegeni`
  * Second database name: `keydb`

For simplicity you can make the database usernames correspond to the database names.

Here's an example of creating a database and a user:

{{{
#!sh
$ mysql -u root -p

mysql> create database seattlegeni;

mysql> GRANT ALL PRIVILEGES 
ON seattlegeni.* 
TO 'seattlegeni'@'localhost'
IDENTIFIED BY 'password';
}}}
where you would replace the 'password' with the password you want to use for the user 'seattlegeni'

'''Repeat the process''', substituting `keydb` for `seattlegeni`. Afterwards, type `\q` to leave the MySQL prompt:

{{{
#!sh
mysql> \q
Bye!
$
}}}



== Deploying and running Clearinghouse ==
In this section, we will deploy and run a copy of the Clearinghouse from your current user account in a temporary directory. This is mainly useful for testing. For an actual deployment, we recommend setting up a separate user account, e.g. named {{{clearinghouse}}}, and following the steps below as this user.

  * '''Initial preparation'''
    * Clone the Clearinghouse repository into your home directory, and let the initialize script fetch dependencies:
{{{
#!sh
git clone https://github.com/aaaaalbert/clearinghouse.git
cd clearinghouse/scripts
python initialize.py
}}}
    * Deploy all necessary files to a directory of your choice. (We'll use `/tmp/TARGET` in these instructions.) The directory you deploy to should be the name of a directory that does not yet exist. For example:
{{{
#!sh 
python ./clearinghouse/deploymentscripts/deploy_seattlegeni.py . /tmp/TARGET
}}} 
    (Note: For a live deployment, the path will likely be {{{/home/geni/live}}}).

    * Deploy the Repy runtime, following the instructions of the script called previously.
{{{
#!sh
mkdir /tmp/TARGET/clearinghouse/seattle
cd clearinghouse/scripts/dist
python build.py /tmp/TARGET/clearinghouse/seattle   # Use absolute target path!
}}}


    * Change to the `clearinghouse` directory that is in the directory you deployed
      to. For example:
{{{
#!sh 
cd /tmp/TARGET/clearinghouse
}}}
      '''Note: All future steps/instructions will assume you are in this directory.'''
  * '''Future tip'''
    * Starting all of the processes described below is simplified by using the script
      `deploymentscripts/start_seattlegeni_components.sh`, whose use is described in
      the section on [#Runningstart_seattlegeni_components.sh running start_seattlegeni_components.sh].
      For reference, we'll explain here how to start the individual components as well
      as include first-time configuration information. Note that when starting these
      manually, you need to start them in the order shown. Most importantly, first the
      lockserver, then the backend, then everything else. However, we're not going to
      describe the initial setup in this order.
  * '''Generating state keys'''
    Seven different state keys need to be created in order for all the backend scripts
    to work. The name of the seven key files should be:
    1. acceptdonation.publickey
    1. movingto_canonical.publickey
    1. canonical.publickey
    1. movingto_onepercentmanyevents.publickey
    1. onepercentmanyevents.publickey
    1. movingto_twopercent.publickey
    1. twopercent.publickey
    * Note that the privatekey of the key pairs is not necessary. To generate the keys 
      follow these instructions: 
      * Navigate to the folder you previously deployed the Repy runtime to, `/tmp/TARGET/seattle`.
      * Run the command:
{{{
python generatekeys.py NAME_OF_YOUR_KEY_FILES 4096
}}}
       This creates a keypair `NAME_OF_YOUR_KEY_FILES.publickey`, `NAME_OF_YOUR_KEY_FILES.privatekey` using a key length of 4096 (the minimum recommended at the time of writing). An example would be:
{{{
python generatekeys.py acceptdonation 4096
}}}
      * Generate a key pair for all state keys per the list above.
      * Create a folder named statekeys in your /tmp/deploy/seattlegeni/node_state_transitions/
        folder and copy the seven publickeys over to that folder.
      * As noted, you can delete the `*.privatekey` files as we won't need them.

  * '''Setup the website and start a development version of it'''
    * Be sure you've already created a mysql database for seattlegeni (e.g.
      called `seattlegeni`).
    * Edit website/settings.py
      * Specify the name of the seattlegeni
        database, as well as the database username, password, etc.
      * Set a long, random string the value of SECRET_KEY.
      * If this is a production launch, also set DEBUG to False,
        and uncomment and change the fields for sending ADMINS email.
    * Set your environment variables:
{{{
#!sh
export PYTHONPATH=$PYTHONPATH:/tmp/deploy:/tmp/deploy/seattle
export DJANGO_SETTINGS_MODULE='seattlegeni.website.settings'
{'fraction_non_alphanumeric': 0.08258670520231214, 'fraction_numerical': 0.0036127167630057803, 'mean_word_length': 3.8142608695652176, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1364226', 'n_tokens_mistral': 3942, 'n_tokens_neox': 3766, 'n_words': 1672}