<jupyter_start><jupyter_text>Ingesting DataThis short notebook ooutlines the steps required to ingest data stored in a Google Sheets doc into the Reforestation Catalogue application. Most steps are automated. Requirements- `pandas='1.1.4'`- `numpy='1.19.5'`- `json='2.0.9'` Steps1. In Google Sheets, download the sheet you wish to export as `csv`: `File --> Download --> 'Comma-separated values (.csv, current sheet)'`.2. Move this file to the local `input` folder at `reforestation-catalogue/data/input`.3. Run the code below to parse the csv and update `mongabay-data.json` in the application with the new data.4. Make a PR with the updated data. UsageThe importCsv function does the following:- Looks inside `/input` folder for `.csv` files. If multiple are present you will be promptd to choose one before continuing iwth the ingestion & update process.- Validates that the expected columns are found. In the case that there are mising columns, the process will fail. If unexpected columns are present in the data, you will be prompted and the process will continue - however these fields will not be recognised insie the application and will not be visible.- Converts column names to slugs as per `./lib/consts/validation_consts.json`- Saves a backup of the previous `mongabay-data.json` file to `/input/mongabay-data-copy.json`.- Overwrites the old `mongabay-data.json` in `public/data/` with new data.<jupyter_code>import project_path
from lib import utils
## Import csv to /public/data/mongabay-data.json and return the validated json
imported_data = utils.importCsv()
## Inspect parsed data
imported_data[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08383961117861483, 'fraction_numerical': 0.009113001215066828, 'mean_word_length': 4.71875, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '25556', 'license': 'MIT', 'n_tokens_mistral': 469, 'n_tokens_neox': 455, 'n_words': 214, 'path': 'data/data-ingress.ipynb', 'repo_name': 'mongabay/reforestation-catalogue', 'size': '1646'}