<jupyter_start><jupyter_text>Train/Test/Validation split of input samples. This notebook shows how train/test/split is being made on a List[InputSample]This is different from the normal split since we don't want sentences generated from the same pattern to be in more than one set. (Applicable only if the dataset was generated from templates)<jupyter_code>from presidio_evaluator import InputSample
from presidio_evaluator.data_generator import read_synth_dataset
from presidio_evaluator.validation import split_dataset, save_to_json
import datetime

%reload_ext autoreload<jupyter_output><empty_output><jupyter_text>Load full dataset<jupyter_code>all_samples = read_synth_dataset("../data/synth_dataset.json")
print(len(all_samples))<jupyter_output><empty_output><jupyter_text>Split to train/test/dev<jupyter_code>TRAIN_TEST_VAL_RATIOS = [0.7,0.2,0.1]

train, test, validation = split_dataset(all_samples,TRAIN_TEST_VAL_RATIOS)<jupyter_output><empty_output><jupyter_text>Train/Test only (no validation)<jupyter_code>#TRAIN_TEST_RATIOS = [0.7,0.3]
#train,test = split_dataset(all_sampleTRAIN_TEST_RATIOSEST_RATIOS)<jupyter_output><empty_output><jupyter_text>Save the different sets to files<jupyter_code>cur_time = datetime.date.today().strftime("%B_%d_%Y")

save_to_json(train,"../data/train_{}.json".format(cur_time))
save_to_json(test,"../data/test_{}.json".format(cur_time))
save_to_json(validation,"../data/validation_{}.json".format(cur_time))

print(len(train))
print(len(test))
print(len(validation))
assert len(train) + len(test) + len(validation) == len(all_samples), "They are not equal"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11479435236341314, 'fraction_numerical': 0.006138735420503376, 'mean_word_length': 9.382165605095542, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '300581', 'license': 'MIT', 'n_tokens_mistral': 573, 'n_tokens_neox': 568, 'n_words': 102, 'path': 'notebooks/Split by pattern #.ipynb', 'repo_name': 'kirkins/presidio-research', 'size': '1629'}