<commit_before># Seattle Clearinghouse Installation How-To

This document explains how to do a fresh install of the Seattle Clearinghouse portal on
a machine that is running a Debian-like Linux operating system such as Ubuntu.

You might want to take a look at the [Seattle Infrastructure Architecture](../../UnderstandingSeattle/SeattleInfrastructureArchitecture.md)

documentation before proceeding. This also mentions the other two
infrastructure components of Seattle Testbed, the Custom Installer
Builder and the software update server, and talk about their interfacing
with the Clearinghouse.

## Setting up a non-privileged user account
First of all, on the server or VM you will be using for the clearinghouse,
we recommend to set up a user account specific to the Clearinghouse
instance you are going to set up. This ensures all of the code, config
files, etc. remain isolated from that of other services on the same machine.

The user *should not be granted interactive login* for security reasons.
Use `sudo -i -u theusername` instead to work in their directory. Needless
to say, the user *should not have root privileges* or be able to acquire them.

Any user name will be fine. We'll use `ch` in the instructions:

```sh
$ sudo adduser ch
```


## Install Dependencies

Clearinghouse requires at least the following software to be installed:
 * [Python](http://www.python.org/) in version 2.7 -- the language Seattle Clearinghouse is written in
 * [Pip](https://pypi.python.org/pypi/pip) -- Recommended tool for installing python packages.
 * [mysqlclient](https://pypi.python.org/pypi/mysqlclient) -- the python mysql interface
 * [MySQL](http://www.mysql.com/) -- the database
 * [Apache 2.4](http://www.apache.org/) -- the web server
 * [mod_wsgi](http://www.modwsgi.org/) -- necessary for interfacing with Django code
 * [OpenSSL](http://www.openssl.org/) -- necessary for `https://` support
 * [Django](http://www.djangoproject.com/) in version 1.8.x -- necessary to run Django code
 * [Django Social Auth](https://github.com/omab/django-social-auth) -- for OpenID and Oauth support.


Most of these can be installed through a package manager. For example, on a Debian-based system:

```sh
$ sudo apt-get install python-pip
$ sudo apt-get install python-dev
$ sudo apt-get install apache2 libapache2-mod-wsgi
$ sudo apt-get install mysql-server mysql-client libmysqlclient-dev
$ # MySQL will prompt you to set a root database password, which you should do.
$ sudo apt-get install ntp
$ sudo apt-get install openssl
```


### Setup python virtualenvironment

A convenient way to isolate your Python installations from each other is
to use `virtualenv`, e.g. if you have specific version requirements for one
project, but don't want to apply them system-wide or for other projects.
[Virtualenv](https://virtualenv.pypa.io/en/latest/index.html) creates a
separate environment that has its own installation directory and that doesn’t
share libraries with other virtualenv environments. The
[virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/)
makes usage of virtualenv even easier. With the following instructions you
can install and setup virtualenv and virtualenvwrapper and create a virtualenv
called `ch` which can be used to install your Seattle Clearinghouse Django
installation.

```sh
$ # Install virtualenv and virtualenvwrapper
$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper

$ # Login as user above created
$ sudo -i -u ch

$ # Create a directory in your home folder where all your virtualenvs will live
$ mkdir ~/.virtualenvs
$ # Make the virtualenvwrapper available
$ echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
$ source ~/.bashrc

$ # Create a new virtualenv called `ch` and immediately switch to it.
$ mkvirtualenv ch
```


In the future you just have to issue the shell command `workon ch` to
switch to the environment or `deactivate` if you want to leave it. You
know that you are *in* the virtualenv when your shell prompt is prefixed by
`(ch)`.
**Note: This prefix is omitted in the shell snippets in the remainder of the document.**



### Install Django

Django, the web framework the clearinghouse uses, is available at https://www.djangoproject.com/download/ or through a package manager. Please note that [SeattleTestbed/clearinghouse:master](https://github.com/seattletestbed/clearinghouse/tree/master) currently supports the 1.8 versions of Django only.
```sh
$ pip install django==1.8.18
```
Depending on your actual OS and setup, this command might require `sudo` privileges.


### Install Mysqlclient
[Mysqlclient](https://pypi.python.org/pypi/mysqlclient) is the
[recommended](https://docs.djangoproject.com/en/1.8/ref/databases/#mysql-db-api-drivers)
choice for using MySQL with Django.

```sh
$ pip install mysqlclient
```

### Install Django Social Auth
Note that even if you don't enable OpenID and OAuth, Clearinghouse requires this specific Django package installed:

```sh
$ pip install django-social-auth
```

## Setup OpenID and OAuth (optional)
If you would like your Clearinghouse to support login not only through user accounts it manages itself, but ID/authentication services like OpenID and OAuth, or web services like Google, Facebook, or GitHub, take a look at the [social auth support instructions page](SocialAuth.md).

## Create MySQL databases

You need two mysql databases and seperate users with access to each.
  * First database name: `clearinghouse`
  * Second database name: `keydb`

For simplicity, make the database usernames correspond to the database names.

Here's an example of creating a database and a user:

```sh
$ mysql -u root -p
$ # This requires entering the database root password set during install!
```
```sql
mysql> create database clearinghouse;

mysql> GRANT ALL PRIVILEGES 
ON clearinghouse.* 
TO 'clearinghouse'@'localhost'
IDENTIFIED BY 'desired password for clearinghouse';

mysql> create database keydb;

mysql> GRANT ALL PRIVILEGES 
ON keydb.* 
TO 'keydb'@'localhost'
IDENTIFIED BY '<PASSWORD> for keydb';

mysql> \q
```
where you would replace the password strings with suitable ones.



## Deploying and running Clearinghouse<commit_msg>CH: Use `sudo` for database setup<commit_after># Seattle Clearinghouse Installation How-To

This document explains how to do a fresh install of the Seattle Clearinghouse portal on
a machine that is running a Debian-like Linux operating system such as Ubuntu.

You might want to take a look at the [Seattle Infrastructure Architecture](../../UnderstandingSeattle/SeattleInfrastructureArchitecture.md)

documentation before proceeding. This also mentions the other two
infrastructure components of Seattle Testbed, the Custom Installer
Builder and the software update server, and talk about their interfacing
with the Clearinghouse.

## Setting up a non-privileged user account
First of all, on the server or VM you will be using for the clearinghouse,
we recommend to set up a user account specific to the Clearinghouse
instance you are going to set up. This ensures all of the code, config
files, etc. remain isolated from that of other services on the same machine.

The user *should not be granted interactive login* for security reasons.
Use `sudo -i -u theusername` instead to work in their directory. Needless
to say, the user *should not have root privileges* or be able to acquire them.

Any user name will be fine. We'll use `ch` in the instructions:

```sh
$ sudo adduser ch
```


## Install Dependencies

Clearinghouse requires at least the following software to be installed:
 * [Python](http://www.python.org/) in version 2.7 -- the language Seattle Clearinghouse is written in
 * [Pip](https://pypi.python.org/pypi/pip) -- Recommended tool for installing python packages.
 * [mysqlclient](https://pypi.python.org/pypi/mysqlclient) -- the python mysql interface
 * [MySQL](http://www.mysql.com/) -- the database
 * [Apache 2.4](http://www.apache.org/) -- the web server
 * [mod_wsgi](http://www.modwsgi.org/) -- necessary for interfacing with Django code
 * [OpenSSL](http://www.openssl.org/) -- necessary for `https://` support
 * [Django](http://www.djangoproject.com/) in version 1.8.x -- necessary to run Django code
 * [Django Social Auth](https://github.com/omab/django-social-auth) -- for OpenID and Oauth support.


Most of these can be installed through a package manager. For example, on a Debian-based system:

```sh
$ sudo apt-get install python-pip
$ sudo apt-get install python-dev
$ sudo apt-get install apache2 libapache2-mod-wsgi
$ sudo apt-get install mysql-server mysql-client libmysqlclient-dev
$ # MySQL will prompt you to set a root database password, which you should do.
$ sudo apt-get install ntp
$ sudo apt-get install openssl
```


### Setup python virtualenvironment

A convenient way to isolate your Python installations from each other is
to use `virtualenv`, e.g. if you have specific version requirements for one
project, but don't want to apply them system-wide or for other projects.
[Virtualenv](https://virtualenv.pypa.io/en/latest/index.html) creates a
separate environment that has its own installation directory and that doesn’t
share libraries with other virtualenv environments. The
[virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/)
makes usage of virtualenv even easier. With the following instructions you
can install and setup virtualenv and virtualenvwrapper and create a virtualenv
called `ch` which can be used to install your Seattle Clearinghouse Django
installation.

```sh
$ # Install virtualenv and virtualenvwrapper
$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper

$ # Login as user above created
$ sudo -i -u ch

$ # Create a directory in your home folder where all your virtualenvs will live
$ mkdir ~/.virtualenvs
$ # Make the virtualenvwrapper available
$ echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
$ source ~/.bashrc

$ # Create a new virtualenv called `ch` and immediately switch to it.
$ mkvirtualenv ch
```


In the future you just have to issue the shell command `workon ch` to
switch to the environment or `deactivate` if you want to leave it. You
know that you are *in* the virtualenv when your shell prompt is prefixed by
`(ch)`.
**Note: This prefix is omitted in the shell snippets in the remainder of the document.**



### Install Django

Django, the web framework the clearinghouse uses, is available at https://www.djangoproject.com/download/ or through a package manager. Please note that [SeattleTestbed/clearinghouse:master](https://github.com/seattletestbed/clearinghouse/tree/master) currently supports the 1.8 versions of Django only.
```sh
$ pip install django==1.8.18
```
Depending on your actual OS and setup, this command might require `sudo` privileges.


### Install Mysqlclient
[Mysqlclient](https://pypi.python.org/pypi/mysqlclient) is the
[recommended](https://docs.djangoproject.com/en/1.8/ref/databases/#mysql-db-api-drivers)
choice for using MySQL with Django.

```sh
$ pip install mysqlclient
```

### Install Django Social Auth
Note that even if you don't enable OpenID and OAuth, Clearinghouse requires this specific Django package installed:

```sh
$ pip install django-social-auth
```

## Setup OpenID and OAuth (optional)
If you would like your Clearinghouse to support login not only through user accounts it manages itself, but ID/authentication services like OpenID and OAuth, or web services like Google, Facebook, or GitHub, take a look at the [social auth support instructions page](SocialAuth.md).

## Create MySQL databases

You need two mysql databases and seperate users with access to each.
  * First database name: `clearinghouse`
  * Second database name: `keydb`

For simplicity, make the database usernames correspond to the database names.

Here's an example of creating a database and a user:

```sh
$ sudo mysql -u root -p
$ # This requires entering the database root password set during install!
```
```sql
mysql> create database clearinghouse;

mysql> GRANT ALL PRIVILEGES 
ON clearinghouse.* 
TO 'clearinghouse'@'localhost'
IDENTIFIED BY 'desired password for clearinghouse';

mysql> create database keydb;

mysql> GRANT ALL PRIVILEGES 
ON keydb.* 
TO 'keydb'@'localhost'
IDENTIFIED BY 'desired password for keydb';

mysql> \q
```
where you would replace the password strings with suitable ones.



## Deploying and running Clearinghouse
{'fraction_non_alphanumeric': 0.08396459612471095, 'fraction_numerical': 0.0025516306514632007, 'mean_word_length': 4.554472984942427, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 22, 'https://': 20, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '511435', 'n_tokens_mistral': 3580, 'n_tokens_neox': 3341, 'n_words': 1572}