<jupyter_start><jupyter_text>The results of the dense coverage of the WSIs in Camelyon17 are at this path:'/mnt/nas4/datasets/ToReadme/PROCESS/denseSampling/uc1-results/'The file system looks like this: INFO.log       l5_c0_p015_n1  l5_c1_p020_n2  l5_c1_p034_n3  l5_c2_p042_n3  l5_c2_p051_n2  l5_c3_p066_n2config.ini     l5_c0_p015_n2  l5_c1_p020_n4  l5_c1_p036_n3  l5_c2_p044_n4  l5_c2_p052_n1  l5_c3_p067_n4l5_c0_p004_n4  l5_c0_p016_n1  l5_c1_p021_n3  l5_c1_p038_n2  l5_c2_p045_n1  l5_c3_p060_n3  l5_c3_p068_n1l5_c0_p009_n1  l5_c0_p017_n1  l5_c1_p022_n4  l5_c1_p039_n1  l5_c2_p046_n3  l5_c3_p061_n4  l5_c3_p072_n0l5_c0_p010_n4  l5_c0_p017_n2  l5_c1_p024_n1  l5_c2_p040_n2  l5_c2_p046_n4  l5_c3_p062_n2  l5_c3_p073_n1l5_c0_p012_n0  l5_c0_p017_n4  l5_c1_p024_n2  l5_c2_p041_n0  l5_c2_p048_n1  l5_c3_p064_n0  l5_c3_p075_n4where the l5_* are folders. The naming structure stands for:l = level of resolution, 5 for allc = center of acquisition, integer from 0 to 3p = patient number, integer from 000 to 100n = node number, integer from 0 to 4Each l5_ * folder has this structure: annotation_mask.png  normal_tissue_mask.png  patches.h5  tumor_locations_0000.pngThe *.png files are just for data validation. annotation_mask.png shows a low-res visualization of the clinicians' annotations of tumor areas. normal_tissue_mask.png shows the non-tumor tissue. tumor_locations_0000.png shows where the patches were extracted from. In this case, since we applied a linear window covering all locations in the WSI, the tumor locations cover every image corner. The file with the image data is patches.h5I will show in the following how to load it and access to the patches<jupyter_code># This is the command to open the file. 
# Very important is to ALWAYS add the 'r' condition to avoid overriding the data!
h5 = h5py.File('/mnt/nas4/datasets/ToReadme/PROCESS/denseSampling/uc1-results/l5_c0_p004_n4/patches.h5', 'r')
# The first file in the data folder is now loaded in the RAM, and we can start inspecting the way it is structured
# We create a list that will contain all the entries in the file. 
# We make this global so that it is accessible from any fuction, anywhere in the code
global dblist
dblist=[]
def list_entries(name, obj):
        if 'patches' in name:
            dblist.append(name)
# visititems is the function to visit all the file entries, the function list_entries will creat the lisdt            
h5.visititems(list_entries) 
# The visititems operation can take a while if the h5 file is large. 
# We now inspect the list of files
print "There are {} entries in the h5py file".format(len(dblist))
print "The patches for the first entry in the file are at this link within the h5py: {}".format(dblist[0])
# The h5py allows to access data as if they were intexed by links. 
# To access some patches, we always have to specify a link within the file
print h5[dblist[0]]
# At the link of dblist[0], there are stored 500 patches of dimension 224x224x3. 
# We now visualize the first patch 
%matplotlib inline
import matplotlib.pyplot as plt
plt.imshow(h5[dblist[0]][0])
# To access the i-th patch, we just have to do h5[dblist[0]][i]
%matplotlib inline
import matplotlib.pyplot as plt
i=50
plt.imshow(h5[dblist[0]][i])
# Another way to inspect the data structure is to ask for the keys with the .keys() function: 
print h5.keys()
print h5['normal'].keys()
print h5['normal']['l5'].keys()
# As you see, here there are quite some keys
# 'normal' and 'tumor' define the tissue type
# 'l5' is fixed for the magnification level
# 'c0' stays for center0, and there could be also 'c1', 'c2', 'c3' and 'c4'
# 'p004' is the patinet number. There are many patients.
# 'n4' is the node number, there are also 'n0', 'n1', 'n2' and 'n3'
print h5['normal']['l5']['c0']['p004']['n4'].keys()
# The keys at h5['normal']['l5']['c0']['p004']['n4'].keys() are all the batches that contain the data for p004. 
# We should access each of them, for example
plt.imshow(h5['normal']['l5']['c0']['p004']['n4']['1310']['patches'][0])
# To have an (intuitive) idea about the patch locations in a batch, you can run
plt.scatter(h5['normal']['l5']['c0']['p004']['n4']['1310']['locations'][:].T[1],h5['normal']['l5']['c0']['p004']['n4']['1310']['locations'][:].T[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09452503509592887, 'fraction_numerical': 0.08844174075807207, 'mean_word_length': 5.004213483146067, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '454936', 'license': 'MIT', 'n_tokens_mistral': 1784, 'n_tokens_neox': 1624, 'n_words': 528, 'path': 'examples/Data_explanation.ipynb', 'repo_name': 'medgift/PROCESS_L2', 'size': '4274'}