<commit_before># Pupa Legistar

Pupa compliant python library for scraping [Legistar sites](http://www.granicus.com/Legistar/Product-Overview.aspx) 
-- legislation management sites hosted by by [Granicus](http://www.granicus.com/Streaming-Media-Government.aspx).

Legistar sites include 
- [Chicago](http://chicago.legistar.com)
- [Philadelphia](http://phila.legistar.com)
- [Oakland](http://oakland.legistar.com/legislation.aspx)
- and many other cities

Pupa framework for managing municipal data
- [standard](https://github.com/opencivicdata/pupa)
- [docs](http://opencivicdata.readthedocs.org/en/latest/)

# Requirements

* python 2.7
* mongo

# Installation

```console
> pip install -r requirements.txt
> python setup.py install 
```

# Setup

Everything should be set up to run already with this repository, but if you want to change your mongo settings, edit `pupa_settings.py`

# Usage

Make sure you have mongo installed and running.

Init the mongo collection. Default database is `pupa`
```console
pupa init chicago 
```

Run the scraper. This will generate a bunch of static json files in `scraped_data` and stuff them in to the `pupa` database.
```console
pupa update chicago
```<commit_msg>less insanely big headers in readme<commit_after># Pupa Legistar

Pupa compliant python library for scraping [Legistar sites](http://www.granicus.com/Legistar/Product-Overview.aspx) 
-- legislation management sites hosted by by [Granicus](http://www.granicus.com/Streaming-Media-Government.aspx).

Legistar sites include 
- [Chicago](http://chicago.legistar.com)
- [Philadelphia](http://phila.legistar.com)
- [Oakland](http://oakland.legistar.com/legislation.aspx)
- and many other cities

Pupa framework for managing municipal data
- [standard](https://github.com/opencivicdata/pupa)
- [docs](http://opencivicdata.readthedocs.org/en/latest/)

## Requirements

* python 2.7
* mongo

## Installation

```console
> pip install -r requirements.txt
> python setup.py install 
```

## Setup

Everything should be set up to run already with this repository, but if you want to change your mongo settings, edit `pupa_settings.py`

## Usage

Make sure you have mongo installed and running.

Init the mongo collection. Default database is `pupa`
```console
pupa init chicago 
```

Run the scraper. This will generate a bunch of static json files in `scraped_data` and stuff them in to the `pupa` database.
```console
pupa update chicago
```
{'fraction_non_alphanumeric': 0.11247947454844007, 'fraction_numerical': 0.0016420361247947454, 'mean_word_length': 4.641203703703703, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4565909', 'n_tokens_mistral': 825, 'n_tokens_neox': 766, 'n_words': 243}