<jupyter_start><jupyter_text>Preprocessing the Dataset<jupyter_code># Python Standard Library
import random
import sys
import os

# External Modules
import numpy as np
import h5py
from datetime import datetime
from skimage.filters import threshold_otsu
from matplotlib import pyplot as plt


from preprocessing.datamodel import SlideManager
from preprocessing.processing import split_negative_slide, split_positive_slide, create_tumor_mask, rgb2gray
from preprocessing.util import TileMap<jupyter_output><empty_output><jupyter_text>DataThe whole data sets have the following sizes:- CAMELYON16 *(~715 GiB)*```data├── CAMELYON16│   ├── training│   │   ├── lesion_annotations│   │   │   └── tumor_001.xml - tumor_110.xml│   │   ├── normal│   │   │   └── normal_001.tif - normal_160.tif│   │   └── tumor│   │       └── tumor_001.tif - tumor_110.tif│   └── test│       ├── lesion_annotations│       │   └── test_001.xml - tumor_110.xml│       └── images│           └── test_001.tif - normal_160.tif```**Note:** For the `SlideManager` class also uppercase and lowercase matters, especially to map annotations to tumor slides, so be consistant in file labeling.   Dataset Generation- Process all files from the CAMELYON16 data set- No overlap for negative tiles- Minimum of 20% tissue in tiles for normal slides- Minimum of 60% tumorours tissue for positive slides- Slide zoom level 0 (0-9, 0 beeing the highest zoom)- Tile_size of 312x312- 156 pixel overlap for tumorous (positive tiles) since they are scarce- We  save up to 1000 tiles per slide- Processing in this notebook will take approximately ~60 hours [\*]- Classifying the tiles of the WSIs of the test set will later take ~1 hour per WSI [\*] Most importantly, we will save all tiles from all WSIs into a single HDF5 file. This is crucial because when accessing the data later for training, most time is consumed when opening a file. Additionally, the training works better, when a single batch (e.g. 100 tiles), is as heterogenous as the original data. So when we want to read 100 random tiles, we ideally want to read 100 tiles from 100 different slides and we do not want to open 100 different files to do so.**Background Information:**Depending on the staining process and the slide scanner, the slides can differ quite a lot in color. Therefore a batch containing 100 tiles from one slide only will most likely prevent the CNN from generalizing well.<jupyter_code>### EDIT THIS CELL:
### Assign the path to your CAMELYON16 data
CAM_BASE_DIR = '/media/nico/data/fourthbrain/project/'
### Do not edit this cell
CAM16_DIR = CAM_BASE_DIR + 'CAMELYON16/'
GENERATED_DATA = CAM_BASE_DIR + 'output/'
mgr = SlideManager(cam16_dir=CAM16_DIR)
n_slides= len(mgr.slides)
level = 0
tile_size = 312
### Execute this cell

poi = 0.20 ### 20% of negative tiles must contain tissue (in contrast to slide background)
poi_tumor = 0.60 ### 60% of pos tiles must contain metastases
### to not have too few positive tile, we use half overlapping tilesize
overlap_tumor = tile_size // 2
### we have enough normal tissue, so negative tiles will be less of a problem
overlap = 0.0
max_tiles_per_slide = 1000<jupyter_output><empty_output><jupyter_text>**Hint:** * As mentioned above, the next two blocks will take alot of time, depending on the choosen option. Before starting to preprocess the full data set it might help to process just a few slides, e.g. two normal and two tumor, to test whether everything works as expected. * In some rare cases jupyter notebook can become unstable when running for hours. It might be a good idea to run the python program from shell instead. To do so export the notebook as python program. Go to `File --> Download as --> Python`  Create Individual Files per WSITo make this process resumable if anything fails, we will first create one HDF5-File for each WSI. This way, if anything fails, like power failure, Python Kernel dying, you can just delete the very last file, which will most likely be corrupted, and resume the process by reexecuting the cells.<jupyter_code>tiles_pos = 0
num_slides = 2 #len(mgr.annotated_slides)

for i in range(num_slides):
    print("Working on {}".format(mgr.annotated_slides[i].name))
#     try: 

    filename = '{}/{}_{}x{}_poi{}_poiTumor{}_level{}.hdf5'.format(GENERATED_DATA, mgr.annotated_slides[i].name, tile_size, tile_size, 
                                                   poi, poi_tumor, level)
    # 'w-' creates file, fails if exists
    h5 = h5py.File(filename, "w-", libver='latest')

    # create a new and unconsumed tile iterator
    tile_iter = split_positive_slide(mgr.annotated_slides[i], level=level,
                                     tile_size=tile_size, overlap=overlap_tumor,
                                     poi_threshold=poi_tumor) 

    tiles_batch = list()
    for tile, bounds in tile_iter:
        if len(tiles_batch) % 10 == 0: print('positive slide #:', i, 'tiles so far:', len(tiles_batch))
        if len(tiles_batch) > max_tiles_per_slide: break
        tiles_batch.append(tile)

    # creating a date set in the file
    dset = h5.create_dataset(mgr.annotated_slides[i].name, 
                             (len(tiles_batch), tile_size, tile_size, 3), 
                             dtype=np.uint8,
                             data=np.array(tiles_batch),
                             compression=0)   
    h5.close()

    tiles_pos += len(tiles_batch)
    print(datetime.now(), i, '/', num_slides, '  tiles  ', len(tiles_batch))
    print('pos tiles total: ', tiles_pos)

#     except:
#         print('slide nr {}/{} failed'.format(i, len(mgr.annotated_slides)))
#         print(sys.exc_info()[0])
tiles_neg = 0
num_slides = 2 #len(mgr.annotated_slides)

for i in range(num_slides):
    try:
        filename = '{}/{}_{}x{}_poi{}_poiTumor{}_level{}.hdf5'.format(GENERATED_DATA, mgr.negative_slides[i].name, tile_size, tile_size, 
                                                       poi, poi_tumor, level)
        # 'w-' creates file, fails if exists
        h5 = h5py.File(filename, "w-", libver='latest')
        
        # load the slide into numpy array
        arr = np.asarray(mgr.negative_slides[i].get_full_slide(level=4))

        # convert it to gray scale
        arr_gray = rgb2gray(arr)

        # calculate otsu threshold
        threshold = threshold_otsu(arr_gray)

        # create a new and unconsumed tile iterator
        # because we have so many  negative slides we do not use overlap
        tile_iter = split_negative_slide(mgr.negative_slides[i], level=level,
                                         otsu_threshold=threshold,
                                         tile_size=tile_size, overlap=overlap,
                                         poi_threshold=poi)

        tiles_batch = []
        for tile, bounds in tile_iter:
            if len(tiles_batch) % 10 == 0: print('neg slide:', i, 'tiles so far:', len(tiles_batch))
            if len(tiles_batch) > max_tiles_per_slide: break
            tiles_batch.append(tile)

        # creating a date set in the file
        dset = h5.create_dataset(mgr.negative_slides[i].name, 
                                 (len(tiles_batch), tile_size, tile_size, 3), 
                                 dtype=np.uint8,
                                 data=np.array(tiles_batch),
                                 compression=0)
        h5.close()
        
        tiles_neg += len(tiles_batch)
        print(datetime.now(), i, '/', num_slides, '  tiles  ', len(tiles_batch))
        print('neg tiles total: ', tiles_neg)
        
    except:
        print('slide nr {}/{} failed'.format(i, len(mgr.negative_slides)))
        print(sys.exc_info()[0])<jupyter_output>neg slide: 0 tiles so far: 0
neg slide: 0 tiles so far: 10
neg slide: 0 tiles so far: 20
neg slide: 0 tiles so far: 30
neg slide: 0 tiles so far: 40
neg slide: 0 tiles so far: 50
neg slide: 0 tiles so far: 60
neg slide: 0 tiles so far: 70
neg slide: 0 tiles so far: 80
neg slide: 0 tiles so far: 90
neg slide: 0 tiles so far: 100
neg slide: 0 tiles so far: 110
neg slide: 0 tiles so far: 120
neg slide: 0 tiles so far: 130
neg slide: 0 tiles so far: 140
neg slide: 0 tiles so far: 150
neg slide: 0 tiles so far: 160
neg slide: 0 tiles so far: 170
neg slide: 0 tiles so far: 180
neg slide: 0 tiles so far: 190
neg slide: 0 tiles so far: 200
neg slide: 0 tiles so far: 210
neg slide: 0 tiles so far: 220
neg slide: 0 tiles so far: 230
neg slide: 0 tiles so far: 240
neg slide: 0 tiles so far: 250
neg slide: 0 tiles so far: 260
neg slide: 0 tiles so far: 270
neg slide: 0 tiles so far: 280
neg slide: 0 tiles so far: 290
neg slide: 0 tiles so far: 300
neg slide: 0 tiles so far: 310
neg slide: 0 tiles [...]<jupyter_text>Create Single FileNow we will create a new, and final HDF5 file to contain all tiles of all WSIs we just created. The benefit of this is to further reduce reading time, as opening a file needs some time and this way we just need to open one single file.<jupyter_code>single_file = '{}/all_wsis_{}x{}_poi{}_poiTumor{}_level{}.hdf5'.format(GENERATED_DATA, tile_size, tile_size, 
                                                       poi, poi_tumor, level)
h5_single = h5py.File(single_file, 'w')
for f in os.listdir(GENERATED_DATA):
    if f.startswith('normal_') or f.startswith('tumor_'):
        filename = GENERATED_DATA + f
        with h5py.File(filename, 'r') as h5:
            for key in h5.keys():
                print('processing: "{}", shape: {}'.format(key, h5[key].shape))
                if h5[key].shape[0] > 0: ### dont create dsets for WSIs with 0 tiles
                    dset = h5_single.create_dataset(key, 
                        h5[key].shape, 
                        dtype=np.uint8,
                        data=h5[key][:],
                        compression=0) 

h5_single.close()
print('Done combining hdf5 files')<jupyter_output>processing: "normal_001", shape: (1001, 312, 312, 3)
processing: "normal_002", shape: (1001, 312, 312, 3)
processing: "tumor_001", shape: (438, 312, 312, 3)
processing: "tumor_002", shape: (31, 312, 312, 3)
{'fraction_non_alphanumeric': 0.09301171332142148, 'fraction_numerical': 0.03335318642048839, 'mean_word_length': 2.823529411764706, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '582503', 'license': 'MIT', 'n_tokens_mistral': 3328, 'n_tokens_neox': 3009, 'n_words': 1155, 'path': 'tiling/create-custom-dataset.ipynb', 'repo_name': 'sfmajors373/FourthBrainBreastCancer', 'size': '10074'}