<commit_before># psync
Parallel Synchronization For Locally Mounted File Systems

Psync is a python library that makes use of Celery for distributing work to
many nodes and processes to achieve the goal of rapid synchronization through
parallelization.

# Requirements
* Python 2.7.10
* [ Redis 3.0.6 ] ( http://redis.io/ )
* [ RabbitMQ ] ( http://www.rabbitmq.com/ )
* [ Python Virtual Env ] ( http://docs.python-guide.org/en/latest/dev/virtualenvs/ )
* (Additional requirements are all python libraries and will be installed below)

# Notes
* All commands are assumed to be run as root unless otherwise specified.

# Installation
* Install rabbitmq
  * Must be accessible to the node that will run the rabbitmq service.  While
    this may seem obvious, a useful setup is to install rabbitmq to a location
    on a shared filesystem, which will allow any node to run the rabbitmq
    service.
  * Contrariwise, if a machine will be dedicated to always run the rabbitmq
    service, the software can be installed local to that machine.
* Install redis
  * Same as for rabbitmq, installing to a shared filesystem will allow any node 
    to run the redis service, or a dedicated machine can have it installed locally.
* Setup python virtual env
  * `pip install virtualenv`
  * `cd my_project_folder`
    * Note: psync should be installed where it is accessible to all worker
      nodes (such as on a shared filesystem).
    * Note: The location is not required to be in any given python path (this
      will be addresed in the psync config file later)
  * `virtualenv venv`
  * `source venv/bin/activate`
* Get pylut (required for psync)
  * `git clone https://github.com/ncsa/pylut.git`
* Get psync
  * `git clone https://github.com/ncsa/psync.git`
* Install the additional python required libraries
  * `cd psync`
  * `pip install -r requirements.txt`
* Edit the config file for your environment
  * `cp config/bashrc.template config/bashrc`
  * `vim config/bashrc`

# Running Psync
There are three major parts of psync that each need to be running before
starting a sync:
1. Message Broker (implements a task queue)
2. Logging Service (centralized logging)
3. Worker Nodes (retrieve tasks from the task queue and execute them)
Each of these services can run on different systems or the same system or any
combination there of.  
The only requirements are that all nodes share a common network and that all
worker nodes mount the necessary filesystems (ie: source, target, and anything
else configured in the bashrc config file).
The rest of this readme will assume that each service will run on a separate node.
It is also assumed that psync is running on a cluster.

1. Start the rabbitmq message broker (task queue)
  1. Login to the machine that will run rabbitmq
  2. `/path/to/psync/bin/rabbitmq_psync start`
  3. `/path/to/psync/bin/rabbitmq_psync status`
  4. If this is the very first time rabbitmq has been run on this node:
    1. `/path/to/psync/bin/rabbitmq_user_setup.sh`
      * This sets up a virtual host and creates a user that can access the
        vhost with the given password.  All these settings are defined in the<commit_msg>Updates to README installation section.<commit_after># psync
Parallel Synchronization For Locally Mounted File Systems

Psync is a python library that makes use of Celery for distributing work to
many nodes and processes to achieve the goal of rapid synchronization through
parallelization.

# Requirements
* Python 2.7.10
* [ Redis 3.0.6 ] ( http://redis.io/ )
* [ RabbitMQ ] ( http://www.rabbitmq.com/ )
* [ Python Virtual Env ] ( http://docs.python-guide.org/en/latest/dev/virtualenvs/ )
* (Additional requirements are all python libraries and will be installed below)

# Notes
* All commands are assumed to be run as root unless otherwise specified.

# Installation
* Install rabbitmq
  * Must be accessible to the node that will run the rabbitmq service.  While
    this may seem obvious, take note that the rabbitmq service does not
    required a dedicated host, it can run from any node. A useful setup is to
    install rabbitmq to a location on a shared filesystem, which will allow any 
    node to run the rabbitmq service.
* Install redis
  * Same as for rabbitmq, installing to a shared filesystem will allow any node 
    to run the redis service, or a dedicated machine can have it installed locally.
* Install Python virtualenv
  * `pip install virtualenv`
* Choose install location.
  * Note: psync should be installed where it will be accessible to all worker
    nodes (such as on a shared filesystem).
  * Note: The location is not required to be in any pre-determined python
    accessible place (this will be addressed in the psync config).
  * This example assumes `$HOME` is sufficient (where the current user is a 
    regular user account, not root)
  * `cd ~` 
* Get pylut (required for psync)
  * `git clone https://github.com/ncsa/pylut.git`
* Get psync
  * `git clone https://github.com/ncsa/psync.git`
* Create a virtualenv for psync
  * `cd psync`
  * `virtualenv venv`
  * `source venv/bin/activate`
* Install the additional python required libraries
  * `pip install -r requirements.txt`
* Edit the config file for your environment
  * `cp config/bashrc.template config/bashrc`
  * `vim config/bashrc`

# Running Psync
There are three major parts of psync that each need to be running before
starting a sync:
1. Message Broker (implements a task queue)
2. Logging Service (centralized logging)
3. Worker Nodes (retrieve tasks from the task queue and execute them)
Each of these services can run on different systems or the same system or any
combination there of.  
The only requirements are that all nodes share a common network and that all
worker nodes mount the necessary filesystems (ie: source, target, and anything
else configured in the bashrc config file).
The rest of this readme will assume that each service will run on a separate node.
It is also assumed that psync is running on a cluster.

1. Start the rabbitmq message broker (task queue)
  1. Login to the machine that will run rabbitmq
  2. `/path/to/psync/bin/rabbitmq_psync start`
  3. `/path/to/psync/bin/rabbitmq_psync status`
  4. If this is the very first time rabbitmq has been run on this node:
    1. `/path/to/psync/bin/rabbitmq_user_setup.sh`
      * This sets up a virtual host and creates a user that can access the
        vhost with the given password.  All these settings are defined in the
{'fraction_non_alphanumeric': 0.0609529702970297, 'fraction_numerical': 0.0049504950495049506, 'mean_word_length': 3.7052401746724892, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3534299', 'n_tokens_mistral': 1844, 'n_tokens_neox': 1763, 'n_words': 912}