<jupyter_start><jupyter_text>Run timings to compare the numpy generator pulling from cache to the tfrecord datasetWhile the numpy generator is super flexible, able to be generated a fresh from a set of DICOMfiles and it will fall back to creating the mask if a cache doesn't exist it is about 50%slower than the tfrecord dataset.Nevertheless, both the tfrecord dataset and the number generator dataset take on the order of10 ms / item for this example with 5 masks.At that time cost, there doesn't seem to be a massive advantage for using the tfrecord.The trade off in loss of flexibility doesn't appear to be worth it.Nevertheless, there may be a significant time difference should the numpy generator need tocopy the data onto the GPU. Potentially the tfrecord dataset may just load the data directlyonto the GPU saving a copy step. Will need to determine experimentally which dataset providesthe best development flexibility / training time trade off. However, I suspect the smalltime reduction that tfrecord provides won't offset the loss in flexibility from using it.<jupyter_code>%%timeit

# Numpy generator dataset
for ct_uid, x_grid, y_grid, input_array, output_array in dataset.take(15):
    pass
%%timeit

# tfrecord dataset
for ct_uid, x_grid, y_grid, input_array, output_array in loaded_dataset.take(15):
    pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03837638376383764, 'fraction_numerical': 0.006642066420664207, 'mean_word_length': 5.191780821917808, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '339041', 'license': 'Apache-2.0', 'n_tokens_mistral': 348, 'n_tokens_neox': 332, 'n_words': 193, 'path': 'prototyping/auto-segmentation/sb/01-attempt-dataset-streaming-from-zenodo/12-moving-into-pymedphys-labs.ipynb', 'repo_name': 'dg1an3/pymedphys', 'size': '1355'}