<commit_before>scraper
=======

Scrapes sites. Gets news. Eventually events.


###Installation

You should probably create a [virtual environment](http://www.virtualenv.org/en/latest/), but
in any event doing `pip install -r requirements.txt` should do the trick. You
might (probably will) have to specify something along the lines of 
`--allow-all-external pattern --allow-unverified pattern` for the pattern
library since it gets downloaded from its homepage. 

The scraper requires  a running MongoDB instance to dump the scraped stories into. 
Make sure you have MongoDB [installed](http://docs.mongodb.org/manual/installation/) 
and type `mongod` at the terminal to begin the instance. MongoDB doesn't require you to prepare
the collection or database ahead of time, so when you run the program it should automatically
create a database called `event_scrape` with a collection called `stories`. Once you've run  `python scraper.py`, 
you can verify that the stories are in the Mongo database by opening a new terminal wind and typing `mongo`. NOTE: 
the scraper runs once every hour, meaning that stories may not immediately appear. To make sure that stories have been 
scraped, comment out the scheduling part at the end of `scraper.py` and uncomment `call_scrape_func`. That will force an
immediate scraping.

From inside Mongo, type `show dbs` to verify that there's a database called `event_scrape`. Type `use event_scrape` and 
`show collections` to make sure there's a `stories` collection. Typing  `db.news_stories.find()` will show you the first 20 entries.

###Running

After everything is installed, it's as simple as `python scraper.py`. That is
assuming, of course, that you wish to use the configuration seen in the
`default_config.ini` file. If not, just modify that. It's only set up to
specify the path to the file of URLs and the MongoDB collection that you want
to used.

The program will run once an hour and will pull from the RSS feeds specified in
the URL file. 

###Contributing

More RSS feeds are always useful. If there's something specific you want to
see, just add it in and open a pull request. The only requirement is that each
line contains one entry with the first column as a unique ID and the second the<commit_msg>Fixed typos in Mongo section<commit_after>scraper
=======

Scrapes sites. Gets news. Eventually events.


###Installation

You should probably create a [virtual environment](http://www.virtualenv.org/en/latest/), but
in any event doing `pip install -r requirements.txt` should do the trick. You
might (probably will) have to specify something along the lines of 
`--allow-all-external pattern --allow-unverified pattern` for the pattern
library since it gets downloaded from its homepage. 

The scraper requires  a running MongoDB instance to dump the scraped stories into. 
Make sure you have MongoDB [installed](http://docs.mongodb.org/manual/installation/) 
and type `mongod` at the terminal to begin the instance. MongoDB doesn't require you to prepare
the collection or database ahead of time, so when you run the program it should automatically
create a database called `event_scrape` with a collection called `stories`. Once you've run  `python scraper.py`, 
you can verify that the stories are in the Mongo database by opening a new terminal window and typing `mongo`. NOTE: 
the scraper runs once every hour, meaning that stories may not immediately appear in your database. To force immediate scraping,
 comment out the scheduling part at the end of `scraper.py` and uncomment `call_scrape_func`. 
 
To interface with Mongo, enter `mongo` at the command line. From inside Mongo, type `show dbs` to verify that there's a database called `event_scrape`. 
Enter the database with `use event_scrape` and type `show collections` to make sure there's a `stories` collection. 
 `db.news_stories.find()` will show you the first 20 entries.

###Running

After everything is installed, it's as simple as `python scraper.py`. That is
assuming, of course, that you wish to use the configuration seen in the
`default_config.ini` file. If not, just modify that. It's only set up to
specify the path to the file of URLs and the MongoDB collection that you want
to used.

The program will run once an hour and will pull from the RSS feeds specified in
the URL file. 

###Contributing

More RSS feeds are always useful. If there's something specific you want to
see, just add it in and open a pull request. The only requirement is that each
line contains one entry with the first column as a unique ID and the second the
{'fraction_non_alphanumeric': 0.060652507116268885, 'fraction_numerical': 0.0008758484782132691, 'mean_word_length': 4.451073985680191, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '153327', 'n_tokens_mistral': 1247, 'n_tokens_neox': 1161, 'n_words': 685}