<commit_before>
# Hosts

## On Duncan.licorn.org

On the `10.0.3.0` network:

- 1     PG/redis/MongoDB (duncan, physical host, production + preview)
- 109    worker (LXC TEMPLATE, must stay down and thus unreachable)
- 110    dev.1flow.net (production, sentry)
- 111    obi (preview, web instance)
- 112    1flow.io (production, web instance)
- 113    worker-01 (production worker, high queue, flower)
- 114    worker-03 (production worker, medium+celery queues, iPython shell)
- 115    worker-05 (production worker, low+celery queues)

## On Gurney.licorn.org

On the `10.0.3.0` network:

- 1     PG/redis/MongoDB (Gurney, physical host, preview + tests)
- 10    worbi (preview worker)
- 11    zero (scratch tests, single node web+worker+flower)
- 12    twenty (idem)
- 113    worker-02 (aux. production worker, low+medium+celery queues)

## On Harvest.licorn.org

On the `10.0.3.0` network:

- 1     host, physical machine
- ??    dev (sentry, has its own PG/redis/memcache)
- 114    worker-04 (aux. production worker, low+medium+celery queues)

# Databases

# Redis

## Configuration

    databases 20

## Numbering and accesses

development & production use same numbers: obviously they are never on the same
machine, there can't be any collision, unlike production & preview which are
on the same machine as of 2013-06-07.

- Main 1flow database (comments, ratings…) is **0**.
    - on preview we could set it to **10**, to avoid mixing prod/preview, except
      if it's a desired behavior (cutting edge code on real data).

- Celery broker and results is **1**.
    - except on preview, where is it **11**. This allows eventually running production and preview on the same Redis server.

- Redis-sessions is **2**,
    - on preview, **12**.

- Test redis database is 9
    - 19 on preview


## Installing a new node

Actions in telegraphic style:

    # OVH: manager
    # edit DNS entries…

    # SSH: duncan
    # edit pg_hba.conf ?
    # edit ~groups/local_config/iptables/iptables.d/* ?

    # SSH: gurney
    sudo lxc-create -n worbi.1flow.io -t ubuntu -- -b olive
    # edit …/config -> lxc.network.ipv4 = …
    # edit /etc/rc.local

    # LOCAL
    # edit .ssh/config … user:olive
    cd ~/Dropbox

    # NOTE: not needed in sparks 2.0
    #fab -H worbi.1flow.io -- sudo apt-get -q update

    fab -H worbi.1flow.io tasks.lxc_base

    # SSH:worbi
    sudo adduser 1flow --force-badname
    sudo adduser 1flow sudo

    # LOCAL
    # edit .ssh/config … user:1flow
    ssh-copy-id worbi
    scp .bashrc worbi:

    # SSH: worbi
    install supervisor

    # LOCAL
    workon 1flow
    cd ~/sources/1flow

    # IF: worker
    # DOESN'T WORK: fab -H worbi.1flow.io test deploy
    fab pick:worbi.1flow.io deploy
<commit_msg>Refresh documentation with sparks 3.5 new features. It's now very easy to deploy a new worker/web node without disturbing the running ones. NICE :-)<commit_after>
# Hosts

## On Duncan.licorn.org

On the `10.0.3.0` network:

- 1     PG/redis/MongoDB (duncan, physical host, production + preview)
- 109    worker (LXC TEMPLATE, must stay down and thus unreachable)
- 110    dev.1flow.net (production, sentry)
- 111    obi (preview, web instance)
- 112    1flow.io (production, web instance)
- 113    worker-01 (production worker, high queue, flower)
- 114    worker-03 (production worker, medium+celery queues, iPython shell)
- 115    worker-05 (production worker, low+celery queues)

## On Gurney.licorn.org

On the `10.0.3.0` network:

- 1     PG/redis/MongoDB (Gurney, physical host, preview + tests)
- 10    worbi (preview worker)
- 11    zero (scratch tests, single node web+worker+flower)
- 12    twenty (idem)
- 113    worker-02 (aux. production worker, low+medium+celery queues)

## On Harvest.licorn.org

On the `10.0.3.0` network:

- 1     host, physical machine
- ??    dev (sentry, has its own PG/redis/memcache)
- 114    worker-04 (aux. production worker, low+medium+celery queues)

# Databases

# Redis

## Configuration

    databases 20

## Numbering and accesses

development & production use same numbers: obviously they are never on the same
machine, there can't be any collision, unlike production & preview which are
on the same machine as of 2013-06-07.

- Main 1flow database (comments, ratings…) is **0**.
    - on preview we could set it to **10**, to avoid mixing prod/preview, except
      if it's a desired behavior (cutting edge code on real data).

- Celery broker and results is **1**.
    - except on preview, where is it **11**. This allows eventually running production and preview on the same Redis server.

- Redis-sessions is **2**,
    - on preview, **12**.

- Test redis database is 9
    - 19 on preview


## Installing a new node

Actions in telegraphic style:

    # OVH: manager
    # edit DNS entries…

    export ENV=production
    export ADMIN=olive
    export WORKER=worker-99.1flow.io

    ssh ${MAIN_SERVER}
    # edit pg_hba.conf ? (if not already OK via global-config)
    # edit ~groups/local_config/iptables/iptables.d/* ? (idem)

    ssh ${LXC_HOST}
    sudo lxc-create -n ${WORKER} -t ubuntu -- -b ${ADMIN}
    # edit …/config -> lxc.network.ipv4 = …
    # edit /etc/rc.local

    vim ~/.ssh/config
    # >>> User olive
    cd ~/sources/global-config

    # NOTE: not needed in sparks 2.0
    #fab -H ${WORKER} -- sudo apt-get -q update

    fab -H ${WORKER} tasks.lxc_base_and_dev

    ssh ${ADMIN}@${WORKER}
    sudo adduser 1flow --force-badname
    # pick a password for the ssh-copy-id
    sudo adduser 1flow sudo

    # LOCAL
    vim ~/.ssh/config
    # >>> "User 1flow"

    ssh-copy-id ${WORKER}
    scp ~/.bashrc ${WORKER}:
    workon 1flow

    # IF: worker
    # DOESN'T WORK: fab -H worbi.1flow.io test deploy
    fab ${ENV} pick:${WORKER} deploy

{'fraction_non_alphanumeric': 0.11028004870412246, 'fraction_numerical': 0.034440772308227514, 'mean_word_length': 2.80794701986755, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6501706', 'n_tokens_mistral': 2112, 'n_tokens_neox': 1897, 'n_words': 684}