<commit_before># Setting Up Celery for NeuroVault

Celery requires a task server. They recommend rabbitmq (and I tested this for the virtual machine) however django (a database) can also be used (but it's not recommended). The other option is redis, but they don't recommend because it is "suseptible to data loss in event of power failures"  This is really just like a task database or queue. I was going to do rabbit, but Gabriel said do redis, so I will do redis. Celery is python based so we install from pip.

     source /opt/nv_env/bin/activate
     pip install -U celery[redis]
     pip install redis

### Installing redis database

     cd /home/vagrant
     wget http://download.redis.io/redis-stable.tar.gz
     tar xvzf redis-stable.tar.gz
     cd redis-stable
     make
     sudo cp src/redis-server /opt/nv_env/local/bin/
     sudo cp src/redis-cli /opt/nv_env/local/bin/

#### Set up places for config and init script

     sudo mkdir /var/redis
     sudo mkdir /etc/redis
     sudo cp utils/redis_init_script /etc/init.d/redis_6379  # default port is 6379

#### change EXEC and CLIEXEC variables for nv_env
     sudo vim /etc/init.d/redis_6379
     EXEC=/opt/nv_env/local/bin/redis-server
     CLIEXEC=/opt/nv_env/local/bin/redis-cli

#### Now copy config file to etc
     sudo cp redis.conf /etc/redis/6379.conf
     sudo vim /etc/redis/6379.conf

     daemonize yes                         # line 37 change no to yes
     pidfile /var/run/redis_6379.pid       # line 41 pidfile
     logfile "/var/log/redis_6379.log"     # line 103, I added logfile
     dir /var/redis/6379                   # line 187, set working / data directory

#### working directory for redis and data
     sudo mkdir /var/redis/6379
     /opt/nv_env/local/bin/redis-server

#### start redis!
Note - I had to use sudo to do this. Not sure if this means potential permissions errors down the line...
     sudo /etc/init.d/redis_6379 start

Test to see if it's working (this is like <NAME>!)
     /opt/nv_env/local/bin/redis-cli ping
     PONG

### Setting up Celery
We need to create a dedicated "celery module" and a celery_tasks script that will hold our different tasks.  The suggestion is to put it in the django top level directory, so the modules get initialized at the same time:

     touch /opt/nv_env/NeuroVault/neurovault/celery.py

The celery.py has a line that will "autodiscover" tasks defined in app directories in a module script named "tasks.py" (the app.autodiscover part below)

     vim /opt/nv_env/NeuroVault/neurovault/apps/statmaps/tasks.py

     from __future__ import absolute_import
<commit_msg>updating readme<commit_after># Setting Up Celery for NeuroVault

Celery requires a task server. They recommend rabbitmq (and I tested this for the virtual machine) however django (a database) can also be used (but it's not recommended). The other option is redis, but they don't recommend because it is "suseptible to data loss in event of power failures"  This is really just like a task database or queue. I was going to do rabbit, but Gabriel said do redis, so I will do redis. Celery is python based so we install from pip.

     source /opt/nv_env/bin/activate
     pip install -U celery[redis]
     pip install redis

### Installing redis database

     cd /home/vagrant
     wget http://download.redis.io/redis-stable.tar.gz
     tar xvzf redis-stable.tar.gz
     cd redis-stable
     make
     sudo cp src/redis-server /opt/nv_env/local/bin/
     sudo cp src/redis-cli /opt/nv_env/local/bin/

#### Set up places for config and init script

     sudo mkdir /var/redis
     sudo mkdir /etc/redis
     sudo cp utils/redis_init_script /etc/init.d/redis_6379  # default port is 6379

#### change EXEC and CLIEXEC variables for nv_env
     sudo vim /etc/init.d/redis_6379
     EXEC=/opt/nv_env/local/bin/redis-server
     CLIEXEC=/opt/nv_env/local/bin/redis-cli

#### Now copy config file to etc
     sudo cp redis.conf /etc/redis/6379.conf
     sudo vim /etc/redis/6379.conf

     daemonize yes                         # line 37 change no to yes
     pidfile /var/run/redis_6379.pid       # line 41 pidfile
     logfile "/var/log/redis_6379.log"     # line 103, I added logfile
     dir /var/redis/6379                   # line 187, set working / data directory

#### working directory for redis and data
     sudo mkdir /var/redis/6379
     /opt/nv_env/local/bin/redis-server

#### start redis!
Note - I had to use sudo to do this. Not sure if this means potential permissions errors down the line...
     
     sudo /etc/init.d/redis_6379 start

Test to see if it's working (this is like <NAME>!)
     
     /opt/nv_env/local/bin/redis-cli ping
     PONG

### Setting up Celery
We need to create a dedicated "celery module" and a celery_tasks script that will hold our different tasks.  The suggestion is to put it in the django top level directory, so the modules get initialized at the same time:

     touch /opt/nv_env/NeuroVault/neurovault/celery.py

The celery.py has a line that will "autodiscover" tasks defined in app directories in a module script named "tasks.py" (the app.autodiscover part below)

     vim /opt/nv_env/NeuroVault/neurovault/apps/statmaps/tasks.py

     from __future__ import absolute_import

{'fraction_non_alphanumeric': 0.07636502481863307, 'fraction_numerical': 0.019091256204658267, 'mean_word_length': 2.824087591240876, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5600955', 'n_tokens_mistral': 1762, 'n_tokens_neox': 1629, 'n_words': 633}