<commit_before>school-report-cards
===================

Tools for parsing annual school report card data from the state of Illinois<commit_msg>added README<commit_after>school-report-cards
===================

Tools for parsing annual school report card data from the state of Illinois

If you run `make` the program will
1. Download ISBE report card data from 1997-2015
2. Convert the fixed with files to csv format, with normalized column names
3. Import the data into normalized tables in a postgresql database

1 and 2 are finished. I'll start with 3 with data that is present
across the entire time period.

## Requirements
* [GNU Make](https://www.gnu.org/software/make/)
* [GNU Wget](https://www.gnu.org/software/wget/)
* [Python](https://www.python.org/downloads/)
* [unoconv](http://dag.wiee.rs/home-made/unoconv/)
* [csvkit](https://csvkit.readthedocs.org/en/0.9.1/install.html)
* [SQLAlchemy](http://www.sqlalchemy.org/)
* [PostgreSQL](http://www.postgresql.org/)
* [unzip](http://www.info-zip.org/)

{'fraction_non_alphanumeric': 0.16023738872403562, 'fraction_numerical': 0.016815034619188922, 'mean_word_length': 4.685393258426966, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6825824', 'n_tokens_mistral': 332, 'n_tokens_neox': 308, 'n_words': 97}