<jupyter_start><jupyter_text>IntroductionAfter the meeting of 28/04/2021, we decided to split the PRIMAP-HIST dataset (with extrapolation) into 2 datasets depending on the 2 different scenarios.The first one will contain all the data with the HISTCR scenario and the other, the HISTTP scenario.  Dataset loading<jupyter_code>import pandas as pd

folder = "../../../data/ghg-emissions/primap-hist/"
dataset_name = "PRIMAP-hist_v2.2_19-Jan-2021.csv"
primap_hist_data = pd.read_csv(folder+dataset_name)
primap_hist_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 28622 entries, 0 to 28621
Columns: 174 entries, scenario to 2018
dtypes: float64(169), object(5)
memory usage: 38.0+ MB<jupyter_text>Dataset splitting Getting all the HISTCR scenario data.<jupyter_code>primap_histcr_data = primap_hist_data[primap_hist_data.scenario == "HISTCR"]
primap_histcr_data.shape
primap_histcr_data.head()
primap_histtp_data = primap_hist_data[primap_hist_data.scenario == "HISTTP"]
primap_histtp_data.shape
primap_histtp_data.head()<jupyter_output><empty_output><jupyter_text>Datasets exports<jupyter_code>primap_histcr_data.to_csv(folder+'PRIMAP-HISTCR.csv')
primap_histtp_data.to_csv(folder+'PRIMAP-HISTTP.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10103420843277645, 'fraction_numerical': 0.03579952267303103, 'mean_word_length': 7.859154929577465, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '57162', 'license': 'MIT', 'n_tokens_mistral': 484, 'n_tokens_neox': 458, 'n_words': 90, 'path': 'notebooks/ghg-emissions/primap-hist/primap-hist-dataset-splitting.ipynb', 'repo_name': 'OpenGeoScales/ogs-data-exploration', 'size': '1257'}