<commit_before># flexicadastre

The ``scraper.py`` scripts is an importer for [FlexiCadastre](http://www.spatialdimension.com/Map-Portals), a GIS solution developed by SpatialDimension and used to store extractives licensing info for a variety of countries.

## Usage

Please set up either Python ``virtualenv`` or Docker on your computer to isolate this project. If you're using ``virtualenv``, the
following should get you up and running:

```bash
$ git clone https://github.com/pudo/flexicadastre.git flexicadastre
$ cd flexicadastre
# Set up the virtualenv (you can also use mkvirtualenv if you have it installed)
$ virtualenv env
# Activate the virtual environment:
$ source env/bin/activate
# Install the dependencies
(env)$ pip install -r requirements.txt
# OPTIONAL: set a permanent DB location
(env)$ export DATABASE_URI="sqlite:///flexicadastre.sqlite"
# Run the actual scraper:
(env)$ python scraper.py
```

The generated GeoJSON and CSV files will be stored in ``data/`` by default, you can set an alternate location using the ``DATA_PATH`` environment variable.

## License

The MIT License (MIT)

Copyright (c) 2015-2017 <NAME>
<commit_msg>simplify code block<commit_after># flexicadastre

The ``scraper.py`` scripts is an importer for [FlexiCadastre](http://www.spatialdimension.com/Map-Portals), a GIS solution developed by SpatialDimension and used to store extractives licensing info for a variety of countries.

## Usage

Please set up either Python ``virtualenv`` or Docker on your computer to isolate this project. If you're using ``virtualenv``, the
following should get you up and running:

```
$ git clone https://github.com/pudo/flexicadastre.git flexicadastre
$ cd flexicadastre
# Set up the virtualenv (you can also use mkvirtualenv if you have it installed)
$ virtualenv env
# Activate the virtual environment:
$ source env/bin/activate
# Install the dependencies
(env)$ pip install -r requirements.txt
# OPTIONAL: set a permanent DB location
(env)$ export DATABASE_URI="sqlite:///flexicadastre.sqlite"
# Run the actual scraper:
(env)$ python scraper.py
```

The generated GeoJSON and CSV files will be stored in ``data/`` by default, you can set an alternate location using the ``DATA_PATH`` environment variable.

## License

The MIT License (MIT)

Copyright (c) 2015-2017 <NAME>

{'fraction_non_alphanumeric': 0.09090909090909091, 'fraction_numerical': 0.006926406926406926, 'mean_word_length': 5.065616797900263, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5743589', 'n_tokens_mistral': 709, 'n_tokens_neox': 664, 'n_words': 285}