<commit_before># RSS Puppy

##### A watchdog tool for monitoring RSS feeds

This tool is designed to monitor RSS feeds in bulk, and to generate machine friendly notifications when new entries appear. While there exists no shortage of RSS readers and web-based notification services, nothing we found combines easy managment of hundreds of RSS feeds with the flexibility to direct output to a variety of data stores or over disparate protocols.

This monitor can be run on any cloud service provider, and requires only Nodejs and a PostgreSQL database. Also, it is trivial to add output handlers which can pipe feed entry data to any service you use.

## How to run

### Get the code

- `git clone https://github.com/buzzfeed-openlab/rss-puppy.git`
- `cd rss-puppy; npm install`
- `cp ./sample-config.json ./config.json`

### Set up a database
The monitor uses a PostgreSQL database to keep track of feeds and entries. 

An easy way to get a reliable, automatically backed up database is to use AWS. Log into the management console, navigate to the `RDS` dashboard, click `Launch a DB instance`, select `PostgreSQL`, and follow the rest of the configuration steps.

Once you have your DB, you can tell the monitor about it using the `config.json` file:

```js
{
    "dbconfig": {
        "user": "DB USER",
        "password": "<PASSWORD>",
        "url": "POSTGRES DB PATH",
        "port": 5432,
        "dbname": "DB NAME",
        "initScript": "./monitor/init-feed-db.sql"
    },
    ...
}<commit_msg>add link to blog post<commit_after># RSS Puppy

##### A watchdog tool for monitoring RSS feeds

This tool is designed to monitor RSS feeds in bulk, and to generate machine friendly notifications when new entries appear. While there exists no shortage of RSS readers and web-based notification services, nothing we found combines easy managment of hundreds of RSS feeds with the flexibility to direct output to a variety of data stores or over disparate protocols.

This monitor can be run on any cloud service provider, and requires only Nodejs and a PostgreSQL database. Also, it is trivial to add output handlers which can pipe feed entry data to any service you use.

[Read more about the motivation and design.](http://www.buzzfeed.com/westleyargentum/your-very-own-rss-watchdog)

## How to run

### Get the code

- `git clone https://github.com/buzzfeed-openlab/rss-puppy.git`
- `cd rss-puppy; npm install`
- `cp ./sample-config.json ./config.json`

### Set up a database
The monitor uses a PostgreSQL database to keep track of feeds and entries. 

An easy way to get a reliable, automatically backed up database is to use AWS. Log into the management console, navigate to the `RDS` dashboard, click `Launch a DB instance`, select `PostgreSQL`, and follow the rest of the configuration steps.

Once you have your DB, you can tell the monitor about it using the `config.json` file:

```js
{
    "dbconfig": {
        "user": "DB USER",
        "password": "<PASSWORD>",
        "url": "POSTGRES DB PATH",
        "port": 5432,
        "dbname": "DB NAME",
        "initScript": "./monitor/init-feed-db.sql"
    },
    ...
}
{'fraction_non_alphanumeric': 0.08620689655172414, 'fraction_numerical': 0.002554278416347382, 'mean_word_length': 3.553779069767442, 'pattern_counts': {'":': 14, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6128971', 'n_tokens_mistral': 888, 'n_tokens_neox': 835, 'n_words': 437}