<commit_before># Installation and configuration of SMC cluster

August 2015

Our SMC cluster will consist of the following:

- db0, db1, db2           -- hostnames of database nodes
- web0, web1              -- hostnames of web server nodes
- compute0, compute1, ... -- hostnames of compute vm's
- admin0, admin1, ...     -- hostnames of admin nodes
- storage                 -- hostname of *the* storage node (yes, single point of failure)

You an configure things so they all run on the same node, but if
you do this and expose it to other users, see the *CRITICAL* db remark below.

## Firewall rules

- the compute vm's should traffic from anything in our cluster
- the web vm's should allow allow traffic from compute (not sure exactly why yet!)
- the db vm's should allow internode db traffic on port 29015
- the db vm's should allow connections from the web nodes on port 28015
- everybody should allow anything from the admin nodes

## All Database/Web Nodes

Settings in /etc/security/limits.conf file:

```
* - memlock unlimited
* - nofile 100000
* - nproc 32768
* - as unlimited
```

## Database Nodes

Configure a clean minimal Ubuntu 15.04 install (db0, db1, ...)


    export H="db0"; gcloud compute --project "sage-math-inc" instances create "$H" --zone "us-central1-c" --machine-type "n1-standard-1" --network "default" --maintenance-policy "MIGRATE" --scopes "https://www.googleapis.com/auth/devstorage.read_write" "https://www.googleapis.com/auth/logging.write" --tags "db" --image "https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1504-vivid-v20150616a" --boot-disk-size "50" --no-boot-disk-auto-delete --boot-disk-type "pd-ssd" --boot-disk-device-name "$H"



This swap leads to horrible pauses and is a very bad idea, unless you have a fast local disk --  set swap space:

    sudo su
    fallocate -l 8G /swapfile && chmod 0600 /swapfile && mkswap /swapfile && swapon /swapfile && echo "/swapfile none swap defaults 0 0" >> /etc/fstab


An assumed account "salvus" to run Rethinkdb as follows:

	sudo su
	apt-get update && apt-get upgrade && apt-get install bup htop fio libprotobuf9 python-pip dstat iotop && pip install rethinkdb && pip3 install rethinkdb && source /etc/lsb-release && echo "deb http://download.rethinkdb.com/apt $DISTRIB_CODENAME main" | sudo tee /etc/apt/sources.list.d/rethinkdb.list && wget -qO- http://download.rethinkdb.com/apt/pubkey.gpg | apt-key add - && apt-get update && apt-get install rethinkdb

    # Configure rethinkdb
    cp /etc/rethinkdb/default.conf.sample /etc/rethinkdb/instances.d/default.conf
    echo "direct-io" >> /etc/rethinkdb/instances.d/default.conf
    echo "bind=all" >> /etc/rethinkdb/instances.d/default.conf
    echo "server-name=`hostname`" >> /etc/rethinkdb/instances.d/default.conf
    #echo "join=db0" >> /etc/rethinkdb/instances.d/default.conf   # careful with this one!
    service rethinkdb restart

If it is a single-site install, don't include the join line above, but<commit_msg>concern about installing python2 version of package<commit_after># Installation and configuration of SMC cluster

August 2015

Our SMC cluster will consist of the following:

- db0, db1, db2           -- hostnames of database nodes
- web0, web1              -- hostnames of web server nodes
- compute0, compute1, ... -- hostnames of compute vm's
- admin0, admin1, ...     -- hostnames of admin nodes
- storage                 -- hostname of *the* storage node (yes, single point of failure)

You an configure things so they all run on the same node, but if
you do this and expose it to other users, see the *CRITICAL* db remark below.

## Firewall rules

- the compute vm's should traffic from anything in our cluster
- the web vm's should allow allow traffic from compute (not sure exactly why yet!)
- the db vm's should allow internode db traffic on port 29015
- the db vm's should allow connections from the web nodes on port 28015
- everybody should allow anything from the admin nodes

## All Database/Web Nodes

Settings in /etc/security/limits.conf file:

```
* - memlock unlimited
* - nofile 100000
* - nproc 32768
* - as unlimited
```

## Database Nodes

Configure a clean minimal Ubuntu 15.04 install (db0, db1, ...)


    export H="db0"; gcloud compute --project "sage-math-inc" instances create "$H" --zone "us-central1-c" --machine-type "n1-standard-1" --network "default" --maintenance-policy "MIGRATE" --scopes "https://www.googleapis.com/auth/devstorage.read_write" "https://www.googleapis.com/auth/logging.write" --tags "db" --image "https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1504-vivid-v20150616a" --boot-disk-size "50" --no-boot-disk-auto-delete --boot-disk-type "pd-ssd" --boot-disk-device-name "$H"



This swap leads to horrible pauses and is a very bad idea, unless you have a fast local disk --  set swap space:

    sudo su
    fallocate -l 8G /swapfile && chmod 0600 /swapfile && mkswap /swapfile && swapon /swapfile && echo "/swapfile none swap defaults 0 0" >> /etc/fstab


An assumed account "salvus" to run Rethinkdb as follows:

hsy: Is there a need for python2 rethinkdb python package?
I think it's not good to install both of them,
because both install those python scripts in /usr/[local?]/bin
and only the python3 version installs clean without an error.

	sudo su
	apt-get update && apt-get upgrade && apt-get install bup htop fio libprotobuf9 python-pip dstat iotop && pip install rethinkdb && pip3 install rethinkdb && source /etc/lsb-release && echo "deb http://download.rethinkdb.com/apt $DISTRIB_CODENAME main" | sudo tee /etc/apt/sources.list.d/rethinkdb.list && wget -qO- http://download.rethinkdb.com/apt/pubkey.gpg | apt-key add - && apt-get update && apt-get install rethinkdb

    # Configure rethinkdb
    cp /etc/rethinkdb/default.conf.sample /etc/rethinkdb/instances.d/default.conf
    echo "direct-io" >> /etc/rethinkdb/instances.d/default.conf
    echo "bind=all" >> /etc/rethinkdb/instances.d/default.conf
    echo "server-name=`hostname`" >> /etc/rethinkdb/instances.d/default.conf
    #echo "join=db0" >> /etc/rethinkdb/instances.d/default.conf   # careful with this one!
    service rethinkdb restart

If it is a single-site install, don't include the join line above, but
{'fraction_non_alphanumeric': 0.11304209183673469, 'fraction_numerical': 0.022480867346938774, 'mean_word_length': 3.411392405063291, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 23, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7262584', 'n_tokens_mistral': 2080, 'n_tokens_neox': 1918, 'n_words': 699}