<commit_before># scraper [![Build Status](https://travis-ci.org/classmere/scraper.svg?branch=master)](https://travis-ci.org/classmere/scraper)

Node.js script scrapes the entire Oregon State course catalog into a PostgreSQL database.

## Usage
Classmere deploys on [Heroku](https://www.heroku.com/home) and follows the standard practice of supplying config data via environment variables. This app looks for the `DATABASE_URL` variable to know which database to connect to. If you don't want to supply your database URL using environment variables, you can supply it via a command line argument:
```shell
npm run -- --db=postgres://<dbuser>:<dbpassword>@<dburl>:<dbport>/<dbname>
```<commit_msg>Updated README.md for Rethink<commit_after># scraper [![Build Status](https://travis-ci.org/classmere/scraper.svg?branch=master)](https://travis-ci.org/classmere/scraper)

Node.js script scrapes the entire Oregon State course catalog into a [RethinkDB](https://www.rethinkdb.com/) database.

## Usage
Classmere deploys on [Heroku](https://www.heroku.com/home) and follows the standard practice of supplying config data via environment variables. This app looks uses four environment variables to configure the RethinkDB connection:
- `DATABASE_HOST`: IP Address of database (default `127.0.0.1`)
- `DATABASE_PORT`: Port of database (default `28015`)
- `DATABASE_DB`: Name of the database (default `test`)
- `DATABASE_KEY`: RethinkDB key (default `''`)
{'fraction_non_alphanumeric': 0.11964038727524205, 'fraction_numerical': 0.007607192254495159, 'mean_word_length': 5.547511312217194, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3121681', 'n_tokens_mistral': 450, 'n_tokens_neox': 427, 'n_words': 146}