<commit_before># ZA Members' Interests Data

There are several files in this directory:

## Raw data

    2010.json
    2011.json
    2012.json
    2013.json
    2014.json

These are the JSON files provided to us by Geoff. They are unchanged and are (I
believe) generated by scraping code that he has from the PDFs mentioned in
them. For me these PDF urls 404ed so I was not able to look at the original
source material.


## Conversion script

    convert_to_import_json.py

This script takes the raw data and processes it to clean it up, to match the mp
to entries in the database and to put it in the format that the
`interests_register_import_from_json` management command expects. This script
is highly specific to the above JSON files and the MP entries in the database
at the time of writing (3 Dec 2013).

You can run the script like this:

    cd pombola/south_africa/data/members-interests/
    ./convert_to_import_json.py 2010.json > 2010_for_import.json
    ./convert_to_import_json.py 2011.json > 2011_for_import.json
    ./convert_to_import_json.py 2012.json > 2012_for_import.json
    ./convert_to_import_json.py 2013.json > 2013_for_import.json
    ./convert_to_import_json.py 2014.json > 2014_for_import.json
    ./convert_to_import_json.py 2015.json > 2015_for_import.json

When processing new data you may well need to add more entries to the
`slug_corrections` attribute. Change the `finding_slug_corrections` to `True`
to enable some code that'll help you do that. Change it back to False when done.

## Final importable data

    2010_for_import.json
    2011_for_import.json
    2012_for_import.json
    2013_for_import.json
    2014_for_import.json
    2015_for_import.json

This is the output of the above conversion script. It is committed for ease of
adding to the database, and as looking at the diffs is an easy way to see the
results of changes to the conversion script.

See `south_africa.md` for the commands needed to load this data into the
database.
<commit_msg>Update README for 2016 + add import instructions<commit_after># ZA Members' Interests Data

There are several files in this directory:

## Raw data

    2010.json
    2011.json
    2012.json
    2013.json
    2014.json
    2015.json
    2016.json

These are the JSON files provided to us by Geoff. They are unchanged and are (I
believe) generated by scraping code that he has from the PDFs mentioned in
them. For me these PDF urls 404ed so I was not able to look at the original
source material.


## Conversion script

    convert_to_import_json.py

This script takes the raw data and processes it to clean it up, to match the mp
to entries in the database and to put it in the format that the
`interests_register_import_from_json` management command expects. This script
is highly specific to the above JSON files and the MP entries in the database
at the time of writing (3 Dec 2013).

You can run the script like this:

    cd pombola/south_africa/data/members-interests/
    ./convert_to_import_json.py 2010.json > 2010_for_import.json
    ./convert_to_import_json.py 2011.json > 2011_for_import.json
    ./convert_to_import_json.py 2012.json > 2012_for_import.json
    ./convert_to_import_json.py 2013.json > 2013_for_import.json
    ./convert_to_import_json.py 2014.json > 2014_for_import.json
    ./convert_to_import_json.py 2015.json > 2015_for_import.json
    ./convert_to_import_json.py 2016.json > 2016_for_import.json

When processing new data you may well need to add more entries to the
`slug_corrections` attribute. Change the `finding_slug_corrections` to `True`
to enable some code that'll help you do that. Change it back to False when done.

## Final importable data

    2010_for_import.json
    2011_for_import.json
    2012_for_import.json
    2013_for_import.json
    2014_for_import.json
    2015_for_import.json
    2016_for_import.json

This is the output of the above conversion script. It is committed for ease of
adding to the database, and as looking at the diffs is an easy way to see the
results of changes to the conversion script.

To load this data into the database, you can run the management command:
(if there are already some entries you should delete them all using the admin)

    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2010_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2011_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2012_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2013_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2014_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2015_for_import.json
    ./manage.py interests_register_import_from_json pombola/south_africa/data/members-interests/2016_for_import.json

{'fraction_non_alphanumeric': 0.05222398729150119, 'fraction_numerical': 0.05003971405877681, 'mean_word_length': 4.077620967741935, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4735600', 'n_tokens_mistral': 1737, 'n_tokens_neox': 1476, 'n_words': 557}