<jupyter_start><jupyter_text>Sketches and progress for SHS I/O<jupyter_code>%matplotlib inline
from __future__ import division, print_function
import numpy as np
import os<jupyter_output>/Users/Jan/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')<jupyter_text>Read a list of all available URI's```Pythondef read_uris():...```<jupyter_code>import SHS_data

uris, ids = SHS_data.read_uris()<jupyter_output><empty_output><jupyter_text>Read cliques```Pythondef read_cliques(clique_file='shs_pruned.txt'):...```<jupyter_code>reload(SHS_data)

cliques_by_name, cliques_by_id = SHS_data.read_cliques()<jupyter_output><empty_output><jupyter_text>Split cliques into train, test & evaluation sets```Pythondef split_train_test_validation(clique_dict, ratio=(50,20,30),                           random_state=1988):...```<jupyter_code>reload(SHS_data)

train_cliques, test_cliques, val_cliques = SHS_data.split_train_test_validation(cliques_by_name)<jupyter_output><empty_output><jupyter_text>Get URI's for a cliqueFirst idea: get URI's and a ground truth matrix.But maybe that's not what we want:18K x 18K ground truth matrix in dense form = 2Gb.Therefore: just URI's for now. Open QuestionShould this function be in SHS_data or somewhere more general?Probably somewhere more general, but there is no such somewhere for now, so leaving it in place.<jupyter_code>reload(SHS_data)

train_uris = SHS_data.uris_from_clique_dict(train_cliques)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09621166566446182, 'fraction_numerical': 0.012026458208057728, 'mean_word_length': 6.961722488038277, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '68164', 'license': 'MIT', 'n_tokens_mistral': 593, 'n_tokens_neox': 562, 'n_words': 134, 'path': 'draft_notebooks/SHS_data_draft.ipynb', 'repo_name': 'jvbalen/cover_id', 'size': '1663'}