<jupyter_start><jupyter_text>data processing to do:- Tulane data needs to rid of out of bound pixels (some contains logo and metadata needs to be painted over) (not if we just use the whole image as training/target)- size of image gonna matter, the context of a local vs big picture can be quite different, especially consideringd the masks are a subset of all channels- we probably have to split image to enrich the dataset. We will see.<jupyter_code>rgb_19_2_wet = data['RGB_19_2_wet']['RGB_19_2_wet']
rgb_19_2_wet.shape
rgb_19_2_wet
plt.imshow(rgb_19_2_wet[1, 0, :, :])
data['TDWB_19_2_chanMaps']['TDWB_19_2_chanMaps'].shape
plt.imshow(data['TDWB_19_2_chanMaps']['TDWB_19_2_chanMaps'][100, :, :])
plt.hist(data['TDWB_19_2_chanMaps']['TDWB_19_2_chanMaps'][100, :, :].ravel())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11028500619578686, 'fraction_numerical': 0.05452292441140025, 'mean_word_length': 7.0, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '85383', 'license': 'MIT', 'n_tokens_mistral': 323, 'n_tokens_neox': 295, 'n_words': 84, 'path': 'notes/data ingestion.ipynb', 'repo_name': 'badGarnet/channel_image_segmentation', 'size': '807'}