<commit_before># Setting up South Africa pombola data

Firstly, download the boundary files from
http://www.demarcation.org.za/Downloads/Boundary/Boundaries.html and unrar
them.

To set up MapIt with these boundaries, you need to create a new MapIt
generation, load in the South Africa fixture, load in the boundaries, then
activate the generation. Hopefully something like this:

    $ python manage.py mapit_generation_create --desc "Initial import" --commit
    $ python manage.py loaddata mapit_south_africa
    $ python manage.py south_africa_import_boundaries --wards=pombola/south_africa/boundary-data/Wards2011.shp --districts=pombola/south_africa/boundary-data/DistrictMunicipalities2011.shp --provinces=pombola/south_africa/boundary-data/Province_New_SANeighbours.shp --locals=pombola/south_africa/boundary-data/LocalMunicipalities2011.shp --commit
    $ python manage.py mapit_generation_activate --commit

Then, to load in some people and organisation data:

    $ python manage.py core_import_popolo pombola/south_africa/data/south-africa-popolo.json  --commit

Run the command to clean up imported slugs:

    $ python manage.py core_list_malformed_slugs --correct

To load in constituency offices run the following.
If you run into any issues with this you might need to remove the
geocode cache at `pombola/south_africa/management/commands/.geocode-request-cache`.

    $ python manage.py south_africa_import_constituency_offices --commit --verbose pombola/south_africa/data/constituencies_and_offices/all_constituencies.csv

To load in some example SayIt data, fetch the speeches/fixtures/test_inputs/

    $ python manage.py load_akomantoso --dir <speeches/fixtures/test_inputs/> --commit

Places can be created to match all those in the mapit database.

    $ ./manage.py core_create_places_from_mapit_entries PRV
    $ ./manage.py core_create_places_from_mapit_entries DMN<commit_msg>Bad slugs no longer seem to be created by the previous import steps<commit_after># Setting up South Africa pombola data

Firstly, download the boundary files from
http://www.demarcation.org.za/Downloads/Boundary/Boundaries.html and unrar
them.

To set up MapIt with these boundaries, you need to create a new MapIt
generation, load in the South Africa fixture, load in the boundaries, then
activate the generation. Hopefully something like this:

    $ python manage.py mapit_generation_create --desc "Initial import" --commit
    $ python manage.py loaddata mapit_south_africa
    $ python manage.py south_africa_import_boundaries --wards=pombola/south_africa/boundary-data/Wards2011.shp --districts=pombola/south_africa/boundary-data/DistrictMunicipalities2011.shp --provinces=pombola/south_africa/boundary-data/Province_New_SANeighbours.shp --locals=pombola/south_africa/boundary-data/LocalMunicipalities2011.shp --commit
    $ python manage.py mapit_generation_activate --commit

Then, to load in some people and organisation data:

    $ python manage.py core_import_popolo pombola/south_africa/data/south-africa-popolo.json  --commit

To load in constituency offices run the following.
If you run into any issues with this you might need to remove the
geocode cache at `pombola/south_africa/management/commands/.geocode-request-cache`.

    $ python manage.py south_africa_import_constituency_offices --commit --verbose pombola/south_africa/data/constituencies_and_offices/all_constituencies.csv

To load in some example SayIt data, fetch the speeches/fixtures/test_inputs/

    $ python manage.py load_akomantoso --dir <speeches/fixtures/test_inputs/> --commit

Places can be created to match all those in the mapit database.

    $ ./manage.py core_create_places_from_mapit_entries PRV
    $ ./manage.py core_create_places_from_mapit_entries DMN
{'fraction_non_alphanumeric': 0.07053500133084908, 'fraction_numerical': 0.006388075592227841, 'mean_word_length': 4.8083462132921175, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3583452', 'n_tokens_mistral': 1221, 'n_tokens_neox': 1123, 'n_words': 329}