<jupyter_start><jupyter_text>OverviewThis notebook will contain the loading component of the Kimberley data loading procedure.In preparation, the original data will have been * uploaded as-is to DPaW's internal CKAN data catalogue, * cleaned in OpenRefine (extract & tranform),* exported as CSV from OpenRefine, and * uploaded as additional resources to the CKAN dataset.This workbook will parse the CSV versions and upload the data to BioSys via its API.Workhorse functions will be located in a separate file `helpers.py`. SetupCopy `secret_template.py` to `secret.py` and modify to contain your CKAN instance and API key.<jupyter_code>import ckanapi
import csv
import json
import requests

from secret import CKAN, LCI, BIOSYS
import helpers as h<jupyter_output><empty_output><jupyter_text>`ck` will be a ckanapi instance that carries your CKAN account's write permissions, and is able to read all public datasets.<jupyter_code>ck = ckanapi.RemoteCKAN(CKAN["dpaw-internal"]["url"], apikey=CKAN["dpaw-internal"]["key"])<jupyter_output><empty_output><jupyter_text>A CKAN resource's URL changes if the file resource changes, but the resource ID will be persistent. The config dict `LCI` lists resource names (from original data worksheet names) against their CKAN resource ID. A helper function `get_data` reads all configured datasets (CSV resources in CKAN).<jupyter_code>data = h.get_data(ck, LCI)
data
[r for r in data["sites"]][0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07513661202185792, 'fraction_numerical': 0.0006830601092896175, 'mean_word_length': 6.288557213930348, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '180539', 'license': 'MIT', 'n_tokens_mistral': 437, 'n_tokens_neox': 414, 'n_words': 177, 'path': 'Kimberley LCI.ipynb', 'repo_name': 'florianm/biosys-etl', 'size': '1464'}