<jupyter_start><jupyter_text>New split datasets now available at :   train_val : `/fast_scratch/WatChMaL/data/IWCDmPMT_4pi_fulltank_9M_splits_CNN/IWCDmPMT_4pi_fulltank_9M_trainval.h5`      test : `/data/WatChMaL/data/IWCDmPMT_4pi_fulltank_9M_splits_CNN/IWCDmPMT_4pi_fulltank_9M_test.h5`<jupyter_code>path = "/fast_scratch/WatChMaL/data/IWCDmPMT_4pi_fulltank_9M_splits_CNN/IWCDmPMT_4pi_fulltank_9M_trainval.h5"
f = h5py.File(path, "r")
print("Keys : ", f.keys())

for key in f.keys():
    print(key, " : ", f[key].shape)
path = "/data/WatChMaL/data/IWCDmPMT_4pi_fulltank_9M_splits_CNN/IWCDmPMT_4pi_fulltank_9M_test.h5"
f = h5py.File(path, "r")
print("Keys : ", f.keys())

for key in f.keys():
    print(key, " : ", f[key].shape)<jupyter_output>Keys :  <KeysViewHDF5 ['angles', 'energies', 'event_data', 'event_ids', 'labels', 'positions']>
angles  :  (3351020, 2)
energies  :  (3351020, 1)
event_data  :  (3351020, 40, 40, 38)
event_ids  :  (3351020,)
labels  :  (3351020,)
positions  :  (3351020, 1, 3)
{'fraction_non_alphanumeric': 0.1536926147704591, 'fraction_numerical': 0.0748502994011976, 'mean_word_length': 5.8231292517006805, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '552240', 'license': 'CNRI-Python', 'n_tokens_mistral': 534, 'n_tokens_neox': 486, 'n_words': 61, 'path': 'notebooks/2211 - New split datasets.ipynb', 'repo_name': 'WatChMaL/CNN', 'size': '1002'}