<commit_before>h2. Introduction

p. Opinoid is a simple web application which syndicates news from a set of known RSS/Atom sources and provides visitors with a sorted list of articles (like Reddit), based on user's votes and comments. It uses "TornadoWeb":http://www.tornadoweb.org/, "MongoDB":http://www.mongodb.org/, "jQuery":http://jquery.com/ and "Bootstrap":http://twitter.github.com/bootstrap/.

p. A live version is available at "alpha.opinoid.com":http://alpha.opinoid.com (hosted on "OpenShift":http://openshift.redhat.com/). Please report issues and ideas here on github!

h2. Hacking

p. The web app file are in the <code>webapp</code> folder, some additional utilities are in the <code>utils</code> folder. The utils initialize the db and and update the list of articles periodically.

h5. webapp

p. The python modules required to get the web app running:

<pre>
$ pip freeze
pep8==0.6.1
pycurl==7.19.0
pygeoip==0.2.2
pymongo==2.0.1
six==1.0.0
textile==2.1.5
tornado==2.1.1
wsgiref==0.1.2
</pre>

p. If you're using python <= 2.6 you will also need to install the <code>argparse</code> module. You'll also need a working install of mongodb.

<pre>
$ ./opinoid.py --help
Usage: ./opinoid.py [OPTIONS]

Options:
  --help                           show this help information
  --log_file_max_size              max size of log files before rollover
  --log_file_num_backups           number of log files to keep
  --log_file_prefix=PATH           Path prefix for log files. Note that if you are running multiple tornado processes, log_file_prefix must be different for each of them (e.g. \
include the port number)
  --log_to_stderr                  Send log output to stderr (colorized if possible). By default use stderr if --log_file_prefix is not set and no other logging is configured.
  --logging=info|warning|error|none Set the Python log level. If 'none', tornado won't touch the logging configuration.
  --dbhost                         DB host
  --dbpass                         DB pass
  --dbport                         DB port
  --dbuser                         DB user
</pre>

p. You will need to run such a *populate.py* script at least once, before starting the web app, to populate the db. We suggest you to set a cronjob to spawn it hourly.

<pre>
15 * * * * /home/release/virtualenv/bin/python /home/release/opinoid/utils/populate.py >> /tmp/populate.log 2>&1
</pre>

h5. utils/populate.py

p. A python script which parses the feeds and populates the db accordingly.

<pre>
$ ./populate.py
2012-04-06 14:15:02,164 INFO STARTED populate
2012-04-06 14:15:02,165 INFO working on http://www.ilfattoquotidiano.it/feed/
2012-04-06 14:15:03,396 INFO entries received 20, skipped 6, added 14
[...]
2012-04-06 14:15:16,024 INFO STARTED cleanup
2012-04-06 14:15:16,025 INFO deleting articles older than 1331036102
2012-04-06 14:15:16,029 INFO articles deleted 1, comments 0
2012-04-06 14:15:16,029 INFO STARTED logrun
2012-04-06 14:15:16,029 INFO last full run at 1333714502
</pre>

p. It will work with any python2 version providing the following modules:

<pre><commit_msg>reports db_host instead of dbhost as command line argument<commit_after>h2. Introduction

p. Opinoid is a simple web application which syndicates news from a set of known RSS/Atom sources and provides visitors with a sorted list of articles (like Reddit), based on user's votes and comments. It uses "TornadoWeb":http://www.tornadoweb.org/, "MongoDB":http://www.mongodb.org/, "jQuery":http://jquery.com/ and "Bootstrap":http://twitter.github.com/bootstrap/.

p. A live version is available at "alpha.opinoid.com":http://alpha.opinoid.com (hosted on "OpenShift":http://openshift.redhat.com/). Please report issues and ideas here on github!

h2. Hacking

p. The web app file are in the <code>webapp</code> folder, some additional utilities are in the <code>utils</code> folder. The utils initialize the db and and update the list of articles periodically.

h5. webapp

p. The python modules required to get the web app running:

<pre>
$ pip freeze
pep8==0.6.1
pycurl==7.19.0
pygeoip==0.2.2
pymongo==2.0.1
six==1.0.0
textile==2.1.5
tornado==2.1.1
wsgiref==0.1.2
</pre>

p. If you're using python <= 2.6 you will also need to install the <code>argparse</code> module. You'll also need a working install of mongodb.

<pre>
$ ./opinoid.py --help
Usage: ./opinoid.py [OPTIONS]

Options:
  --help                           show this help information
  --log_file_max_size              max size of log files before rollover
  --log_file_num_backups           number of log files to keep
  --log_file_prefix=PATH           Path prefix for log files. Note that if you are running multiple tornado processes, log_file_prefix must be different for each of them (e.g. \
include the port number)
  --log_to_stderr                  Send log output to stderr (colorized if possible). By default use stderr if --log_file_prefix is not set and no other logging is configured.
  --logging=info|warning|error|none Set the Python log level. If 'none', tornado won't touch the logging configuration.
  --db_host                        DB host
  --db_pass                        DB pass
  --db_port                        DB port
  --db_user                        DB user
</pre>

p. You will need to run such a *populate.py* script at least once, before starting the web app, to populate the db. We suggest you to set a cronjob to spawn it hourly.

<pre>
15 * * * * /home/release/virtualenv/bin/python /home/release/opinoid/utils/populate.py >> /tmp/populate.log 2>&1
</pre>

h5. utils/populate.py

p. A python script which parses the feeds and populates the db accordingly.

<pre>
$ ./populate.py
2012-04-06 14:15:02,164 INFO STARTED populate
2012-04-06 14:15:02,165 INFO working on http://www.ilfattoquotidiano.it/feed/
2012-04-06 14:15:03,396 INFO entries received 20, skipped 6, added 14
[...]
2012-04-06 14:15:16,024 INFO STARTED cleanup
2012-04-06 14:15:16,025 INFO deleting articles older than 1331036102
2012-04-06 14:15:16,029 INFO articles deleted 1, comments 0
2012-04-06 14:15:16,029 INFO STARTED logrun
2012-04-06 14:15:16,029 INFO last full run at 1333714502
</pre>

p. It will work with any python2 version providing the following modules:

<pre>
{'fraction_non_alphanumeric': 0.10046541486117798, 'fraction_numerical': 0.06419515326592842, 'mean_word_length': 3.1135313531353135, 'pattern_counts': {'":': 12, '<': 35, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5344811', 'n_tokens_mistral': 2222, 'n_tokens_neox': 1928, 'n_words': 757}