<commit_before>Atifier
--------

Generate RSS/Atom feed from changes in or in a specific region of a web page.

Installation
-------------

There are two dependencies for the project:
* django v1.9+
* node.js v4.4.0+

Run the following steps to install the server:

```
$ virtualenv at_env
$ source ~/at_env/bin/activate
$ git clone https://github.com/theju/atifier.git
$ cd atifier/server/
$ pip install -r requirements.txt
$ touch server/local.py
# Add the database settings and install the db driver
$ python manage.py migrate
$ python manage.py createsuperuser
$ python manage.py runserver
```

For the scraper:<commit_msg>Updated readme with a few more details.<commit_after>Atifier
--------

Generate RSS/Atom feed from changes in or in a specific region of a web page.

The project has two components: a server and scraper.

The server is powered by the django admin:

  - stores a list of URLs along with the interval, DOM selector to be scraped
  - the last computed hash (SHA1) of the DOM selector (region of interest)
  - generates RSS/Atom feeds based on the content-type

The scraper uses PhantomJS:

  - Fetches the page and evaluates against a DOM selector (passed to document.querySelector)
  - Get the innerHTML of the DOM selector and returns it's SHA1 hash to the server

Installation
-------------

There are two dependencies for the project:
* django v1.9+
* node.js v4.4.0+

Run the following steps to install the server:

```
$ virtualenv at_env
$ source ~/at_env/bin/activate
$ git clone https://github.com/theju/atifier.git
$ cd atifier/server/
$ pip install -r requirements.txt
$ touch server/local.py
# Add the database settings and install the db driver
$ python manage.py migrate
$ python manage.py createsuperuser
$ python manage.py runserver
```

For the scraper:
{'fraction_non_alphanumeric': 0.09308807134894091, 'fraction_numerical': 0.006688963210702341, 'mean_word_length': 3.5558375634517767, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5901785', 'n_tokens_mistral': 532, 'n_tokens_neox': 515, 'n_words': 230}