<commit_before>UWMadCrawler
============

A Scrapy spider script to crawl the UW-Madison course register


## How To Use
This program uses Scrapy, an open source web scraping library designed using Python. To learn more, check out the [Docs](http://doc.scrapy.org/en/latest/intro/overview.html)

To run the crawler, [install Scrapy](http://doc.scrapy.org/en/latest/intro/install.html), then run the command 
> scrapy crawl UWMad

This will crawl the website and print out the courses on the register.

## Notice
This program is in heavy beta, as are the docs. Please stay tuned for updates and updated documentation.





<commit_msg>Add requirements installation to readme<commit_after>UWMadCrawler
============

A Scrapy spider script to crawl the UW-Madison course register


## How To Use
This program uses [Scrapy](http://scrapy.org/), an open source web scraping library designed using Python. To learn more, check out the [Docs](http://doc.scrapy.org/en/latest/intro/overview.html)

First, install the project's dependencies with pip:

    pip install -r requirements.txt

Then run the following command in the UWMadCrawler/ directory to run the crawler:

    cd UWMadCrawler
    scrapy crawl UWMad

This will crawl the website and print out the courses on the register.

## Notice
This program is in heavy beta, as are the docs. Please stay tuned for updates and updated documentation.






{'fraction_non_alphanumeric': 0.08220157255182273, 'fraction_numerical': 0.0, 'mean_word_length': 4.223880597014926, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '388617', 'n_tokens_mistral': 422, 'n_tokens_neox': 378, 'n_words': 178}