<commit_before># PowerToken Background Scripts
### (Last Update, 3/16/2018)


## Dependencies

* Python 2.7 (but should be compatible with Python 3)
* Python Requests 2.18.4 (http://docs.python-requests.org/en/master/)
* Python SQLAlchemy 1.2.5 (http://docs.sqlalchemy.org/en/latest)
* Fitbit Web API (https://dev.fitbit.com/reference/web-api/quickstart/)
* WEconnect Web API (documentation not available to the public)


## Brief Overview

The scripts [maintenance.py] and [polling.py] are meant to be run as Cron jobs. You can edit your Cron Tab to run them periodically using `crontab -e` and adding these two lines:

`0 * * * * bash <absolute-path>/powertoken/run_maintenance.sh`
`0,15,30,45 * * * * bash <absolute-path>/powertoken/run_polling.sh`

We have the maintenance script set to run at the beginning of every hour and the polling script to run at the 0, 15, 30, and 45 minute marks (effectively every 15 minutes).

You might notice that we are not running [maintenance.py] and [polling.py] directly from Cron. This is because both utilize a virtualenv, which Cron has no knowledge of. Instead, the bash scripts [run_maintenance.sh] and [run_polling.sh] activate the virtualenv, run the respective Python script, and then deactivate the virtualenv.


## Database Maintenance

Every hour, the [maintenance.py] script performs the following activities:

In the `users` table of the database:
* Makes sure all user fields are complete, and removes incomplete profiles.
* Makes sure all WEconnect and Fitbit access tokens are unexpired.

In the `activities` table:
* If any users have been removed from the database, deletes their activity records.
* Removes any expired activities.
* If users have added or updated activities, adds those to the database.

In the `logs` table:
* If any users have been removed from the database, deletes their logs.


## Poll WEconnect and update Fitbit

The [polling.py] script performs the main function of the PowerToken system. The workflow is as follows:

1. If any activities (from the database) start or end within the next 15 minutes, add the user who owns them to a list. <commit_msg>Fixed incorrect MD format.<commit_after># PowerToken Background Scripts
### (Last Update, 3/16/2018)


## Dependencies

* Python 2.7 (but should be compatible with Python 3)
* Python Requests 2.18.4 (http://docs.python-requests.org/en/master/)
* Python SQLAlchemy 1.2.5 (http://docs.sqlalchemy.org/en/latest)
* Fitbit Web API (https://dev.fitbit.com/reference/web-api/quickstart/)
* WEconnect Web API (documentation not available to the public)


## Brief Overview

The scripts [maintenance.py](maintenanc.py) and [polling.py](polling.py) are meant to be run as Cron jobs. You can edit your Cron Tab to run them periodically using `crontab -e` and adding these two lines:

`0 * * * * bash <absolute-path>/powertoken/run_maintenance.sh`
`0,15,30,45 * * * * bash <absolute-path>/powertoken/run_polling.sh`

We have the maintenance script set to run at the beginning of every hour and the polling script to run at the 0, 15, 30, and 45 minute marks (effectively every 15 minutes).

You might notice that we are not running [maintenance.py](maintenanc.py) and [polling.py](polling.py) directly from Cron. This is because both utilize a virtualenv, which Cron has no knowledge of. Instead, the bash scripts run_maintenance.sh and run_polling.sh activate the virtualenv, run the respective Python script, and then deactivate the virtualenv.


## Database Maintenance

Every hour, the [maintenance.py](maintenance.py) script performs the following activities:

In the `users` table of the database:
* Makes sure all user fields are complete, and removes incomplete profiles.
* Makes sure all WEconnect and Fitbit access tokens are unexpired.

In the `activities` table:
* If any users have been removed from the database, deletes their activity records.
* Removes any expired activities.
* If users have added or updated activities, adds those to the database.

In the `logs` table:
* If any users have been removed from the database, deletes their logs.


## Poll WEconnect and update Fitbit

The [polling.py](polling.py) script performs the main function of the PowerToken system. The workflow is as follows:

1. If any activities (from the database) start or end within the next 15 minutes, add the user who owns them to a list. 
{'fraction_non_alphanumeric': 0.0809818765771966, 'fraction_numerical': 0.016976370727231016, 'mean_word_length': 4.456821026282854, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5354377', 'n_tokens_mistral': 1307, 'n_tokens_neox': 1200, 'n_words': 583}