<commit_before># Canadian Legislative Scrapers

See [blank-pupa](https://github.com/opennorth/blank-pupa) to install dependencies and get started.

## Run a scraper

    python -m pupa.cli update --nonstrict ca_ab_edmonton

To run only the scraping step and skip the import step into MongoDB add the `--scrape` switch:

    python -m pupa.cli update --nonstrict --scrape ca_ab_edmonton

For documentation on the `pupa.cli` command:

    python -m pupa.cli -h

For documentation on the `update` subcommand:

    python -m pupa.cli update -h

## Create a scraper

Find division identifiers using the [Open Civic Data Division Identifier (OCD-ID) Viewer](http://opennorth.github.io/ocd-id-viewer/) or by browsing the [list of identifiers](https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-ca.csv). In most cases, a municipality will have a division identifier with a type ID of `csd`. Then, create a scraper with:

    invoke new --division-id ocd-division/country:ca/csd:5915022

This command creates an `__init__.py` file and a stub `people.py` file within a new directory for the scraper. The `__init_.py` file, which describes the jurisdiction, should not require any editing.

Most jurisdictions have a `geographic_code` that corresponds to a [Standard Geographical Classification (SGC) 2011](http://www.statcan.gc.ca/subjects-sujets/standard-norme/sgc-cgt/2011/sgc-cgt-intro-eng.htm) code. Other jurisdictions have a `division_id` that corresponds to am [OCD-ID](https://github.com/opencivicdata/ocd-division-ids).

## Develop a scraper

### Troubleshooting

If the `pupa.cli` command raises the error below, ensure that MongoDB is running.
<commit_msg>correct typo<commit_after># Canadian Legislative Scrapers

See [blank-pupa](https://github.com/opennorth/blank-pupa) to install dependencies and get started.

## Run a scraper

    python -m pupa.cli update --nonstrict ca_ab_edmonton

To run only the scraping step and skip the import step into MongoDB add the `--scrape` switch:

    python -m pupa.cli update --nonstrict --scrape ca_ab_edmonton

For documentation on the `pupa.cli` command:

    python -m pupa.cli -h

For documentation on the `update` subcommand:

    python -m pupa.cli update -h

## Create a scraper

Find division identifiers using the [Open Civic Data Division Identifier (OCD-ID) Viewer](http://opennorth.github.io/ocd-id-viewer/) or by browsing the [list of identifiers](https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-ca.csv). In most cases, a municipality will have a division identifier with a type ID of `csd`. Then, create a scraper with:

    invoke new --division-id ocd-division/country:ca/csd:5915022

This command creates an `__init__.py` file and a stub `people.py` file within a new directory for the scraper. The `__init_.py` file, which describes the jurisdiction, should not require any editing.

Most jurisdictions have a `geographic_code` that corresponds to a [Standard Geographical Classification (SGC) 2011](http://www.statcan.gc.ca/subjects-sujets/standard-norme/sgc-cgt/2011/sgc-cgt-intro-eng.htm) code. Other jurisdictions have a `division_id` that corresponds to an [OCD-ID](https://github.com/opencivicdata/ocd-division-ids).

## Develop a scraper

### Troubleshooting

If the `pupa.cli` command raises the error below, ensure that MongoDB is running.

{'fraction_non_alphanumeric': 0.09718518518518518, 'fraction_numerical': 0.008888888888888889, 'mean_word_length': 4.3081761006289305, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1705160', 'n_tokens_mistral': 1113, 'n_tokens_neox': 1024, 'n_words': 371}